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Introduction 

This  project  is  broken  into  three  focus  areas:  robotic  curriculum,  telesurgery,  and  simulation.  In  each  we 
are  exploring  various  applications  and  extensions  of  the  existing  robotic  surgical  systems.  Under  robotic 
curriculum  we  are  bringing  together  the  leading  surgeons  and  academicians  to  define  the  outcomes 
measures,  curriculum,  and  high  stakes  testing  that  should  be  used  to  certify  surgeons  who  wish  to  practice 
robotic  surgery.  Under  telesurgery  we  are  exploring  the  ability  to  perform  telesurgery  using  a  robot  within 
a  metropolitan  area  based  on  the  currently  available  technology.  Under  simulation  we  are  examining  the 
impact  of  rehearsing  a  procedure  in  a  simulator  immediately  before  performing  that  same  procedure  on  a 
patient.  This  area  also  includes  a  comparative  evaluation  of  all  of  the  robotic  simulators  that  are  available 
with  a  recommendation  of  the  best  fit  for  military  surgeons. 
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Statement  of  Work 


ORIGINAL  STATEMENT  OF  WORK 

There  are  three  primary  areas  of  this  research:  Telesurgery,  Simulation,  and  Robotic  Curriculum.  (1)  The 
telesurgery  project  will  identify  the  characteristics  of  latency  during  telesurgery  and  investigate  the 
application  of  principles  of  automatic  surgery.  (2)  Under  simulation,  we  will  validate  a  simulator  that  can 
be  used  by  military  surgeons  to  maintain  their  robotic  skills  while  deployed.  We  will  then  use  this  device 
to  explore  the  feasibility  of  surgical  rehearsal  as  a  potential  solution  to  the  latency  issue  in  telesurgery.  (3) 
We  will  organize  robotic  surgery  experts  to  develop  a  nationally  accepted  curriculum  in  the  Fundamentals 
of  Robotic  Surgery  (FRS). 

Period  1 

Telesurgery:  Communications  Latency  Experiments.  Identify  communication  latency,  measure  safe 
latency  levels  for  each  robotic  movement,  modify  surgical  procedures  to  be  effective  in  this  environment. 
Milestone:  Telesurgery  latency  experiment  report.  Award  +  270  days 

Simulation:  Military-use  Validation.  Validate  a  robotic  simulator  for  maintaining  the  robotic  surgery 
skills  of  deployed  military  surgeons. 

Milestone:  Robotic  simulator  validation  report.  Award  +  210  days 

Robotic  Curriculum:  Consensus  Conferences.  Organize  and  host  conferences  of  approximately  40  leading 
robotic  surgeons  from  around  the  United  States  to  include  military  surgeons.  Identify  the  fundamental 
knowledge  and  skills  that  should  be  a  foundation  for  every  robotic  surgeon. 

Milestone:  FRS  consensus  conference  reports.  Award  +180  days  and  365  days 

Period  2 

Telesurgery:  Automatic  Surgery.  Apply  movements  recorded  in  a  robotic  simulator  to  actual  execution 
with  the  da  Vinci  robot  on  solid  models.  Explore  ability  to  automatically  execute  surgery  from  a  simulator 
recording. 

Milestone:  Automatic  surgery  experiment  results.  Award  +  730  days 

Simulation:  Surgical  Rehearsal.  Experiment  with  the  effectiveness  of  simulated  surgical  rehearsal  on 
improving  the  outcomes  of  robotic  surgery. 

Milestone:  Surgical  rehearsal  experiment  results.  Award  +  540  days 

FRS  Curriculum  Validation  and  Transition,  Develop  specific  training  tasks  and  passing  criteria  for  the 
FRS  curriculum.  Process  the  curriculum  through  the  certifying  bodies. 

Milestone:  Telesurgery  medical  procedure  results.  Award  +  730  days 
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Executive  Summary 

Telesurgery:  Communications  Latency  Experiments.  As  of  August  30,  2012,  84  subjects  have  participated 
in  this  experiment.  We  continue  to  collect  subject  data.  The  data  collected  in  the  experiment  has  shed 
light  on  a  number  of  details  around  robotic  surgery.  In  general  we  find  that  the  effect  of  latency  on 
individual  surgeons  is  not  predictable  by  their  levels  of  experience  in  either  robotics  or  laparoscopy.  The 
performance  under  latency  varies  widely  at  all  latency  settings  and  all  levels  of  surgical  experience.  The 
majority  of  subjects,  though  not  all,  can  manage  latency  at  200ms  and  below.  Above  this,  most  subjects  of 
all  experience  levels  exhibit  severe  drops  in  performance.  Previous  research  projects  have  suggested  that 
latency  below  250ms  could  be  considered  manageable  because  it  is  below  the  perception  threshold  of 
most  people.  Our  experiments  agree  with  this  for  a  majority  of  subjects,  though  certainly  not  for  all. 

The  data  collected  during  this  experiment  also  allowed  us  to  study  the  relationship  between  years  of 
laparoscopic  experience  and  performance  in  a  robotic  environment.  We  found  a  consistent  negative 
correlation  between  these  two  variables.  It  appears  that  extensive  laparoscopic  experience  is  detrimental 
to  the  acquisition  of  skills  in  robotic  surgery. 

Papers  on  both  of  these  results  have  been  submitted  for  presentation  at  conferences  and  are  being 
prepared  for  journal  publication. 

Simulation:  Military-use  Validation.  At  the  project  kick-off  of  this  grant  in  September,  2011  we  requested 
to  move  this  experiment  to  year  two  of  the  grant.  This  was  motivated  by  the  lead  time  to  purchase  the 
necessary  equipment.  This  change  was  accepted  by  the  government  and  this  project  will  be  performed  in 
the  second  year. 

Robotic  Curriculum:  Consensus  Conferences.  We  have  held  three  conferences  of  leading  robotic 
surgeons  from  around  the  world.  Eighteen  participated  in  the  outcomes  measures  meeting,  forty 
participated  in  the  curriculum  development  meeting,  and  twenty-four  participated  in  the  curriculum 
writing  meeting.  We  have  identified  a  list  of  25  outcomes  measures  that  robotic  surgeons  need  to  be  able 
to  demonstrate.  Three  different  curriculums  have  been  created  -  didactic,  psychomotor  skills,  and  team 
training.  A  multi-skills  device  for  testing  many  of  the  outcomes  has  been  designed  and  will  be  produced 
in  the  second  year. 

We  are  in  discussions  with  SAGES  to  develop  a  High  Stakes  Test  for  this  material  and  to  create  a  system 
to  administer  that  test. 

Simulation:  Surgical  Rehearsal.  At  the  government  kick-off  meeting  we  requested  to  move  this 
experiment  to  year  one  in  place  of  the  military-use  validation  study.  Approval  for  that  change  was  given 
by  the  government.  Multiple  protocols  were  designed  for  conducting  this  experiment;  however,  none  of 
them  were  both  feasible  for  implementation  and  acceptable  to  the  government  reviewers.  A  new  protocol 
is  in  draft  now  and  will  be  presented  to  the  government  for  review  at  the  end  of  September  2012.  A  brief 
summary  of  that  protocol  is  included  in  this  report,  but  has  not  been  completed  or  approved  at  this  time. 
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Telesurgery:  Communications  Latency 


This  experiment  investigates  the  impact  of  communication  latency  on  robotic  surgical  performance, 
where  latency  is  quantitatively  defined  as  length  of  time  (in  milliseconds)  for  the  computer  command 
from  a  surgeon"s  hand  movement  to  be  transmitted  to  the  robot  end  effectors,  and  for  the  image  of  that 
movement  to  return  to  the  surgeon"s  video  display  system.  In  this  experiment  we  are  interested  in  the 
effect  that  latency  has  on  the  completion  time  of  the  entire  surgical  tasks  presented  and  the  effect  on  the 
number  of  errors  that  are  induced  on  the  patient. 

This  is  a  prospective,  randomized,  observational  study.  We  simulated  different  communication  latencies 
ranging  from  0  to  1000  milliseconds  (ms)  sequentially  in  100  ms  increments.  We  randomly  assigned  a 
different  latency  to  each  subject  for  each  trial.  In  order  to  avoid  repetition,  the  randomized  latency  used  a 
block  design  so  that  latency  values  were  all  used  before  the  set  is  used  again.  This  assignment  insured 
that:  1)  each  subject  received  a  random  latency  for  their  exercises,  and  2)  each  latency  level  has  roughly 
equal  number  of  observations  (i.e.  the  observations  will  equally  cover  all  situations). 

One  trial  consisted  of  the  following  sequence  of  activities: 

1 .  Acquaintance  Period.  The  subjects  walked  through  the  curriculum  of  4  tasks  on  the 
robot/simulator.  They  were  allowed  to  get  acquainted  with  the  simulator  controls. 

2.  Baseline  Case.  The  subject  performed  a  single  complete  trial  with  no  latency  in  the 
system.  The  zero  latency  (0  ms)  case  provided  a  baseline  performance  of  the  subject  for 
later  comparison.  It  also  provided  a  set  of  data  useful  for  analyzing  performance  in  a 
standard  robotic  environment. 

3.  Short  term  adaptation.  Prior  studies  (Rayman  et  al  2005,  2006),  have  shown  that  subjects 
improve  their  performance  in  a  latency  environment  over  three  different  repetitions 
before  plateauing  at  their  long-term  performance  level  on  the  fourth  repetition.  We 
selected  the  latency  level  for  the  subject  and  allow  them  to  perform  four  tasks  to  move 
past  the  short-term  learning  curve  phase. 

4.  Trial  Case.  Finally,  we  allowed  the  subject  to  perform  the  specified  tasks  that  compose  a 
single  trial.  This  repetition  is  their  long-term  performance  level.  This  trial  is  the  one  that 
is  included  in  the  statistical  analysis  of  the  study. 

Experimental  Tools 

The  latency  effect  is  created  using  the  dV-Trainer  simulator  (Figure  1)  of  the  da  Vinci  surgical  robot 
(Hung,  2011;  Kennedy  2009).  The  simulator  allows  the  insertion  of  specific  levels  of  controlled  latency 
so  that  the  user"s  physical  movements  are  not  manifest  by  the  simulated  instruments  until  after  the 
defined  latency  period  has  elapsed.  The  simulator  does  not  have  the  ability  to  separately  define  the 
latency  of  messages  traveling  surgeon-to-robot  and  robot-to-surgeon. 
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Figure  1.  dV-Trainer  Simulator  (Mimic  Technologies,  Inc.) 

During  actual  telesurgery,  the  messages  sent  between  the  surgeon's  machine  and  the  remote  patient  station 
will  be  delayed  due  to  the  speed  of  light  and  the  message  routing  that  occurs  on  the  internet.  Determining 
how  much  latency  can  be  safely  tolerated  in  surgery  is  an  important  question  (Anvari,  2005  and  2007). 
This  experiment  hypothesizes  that  there  are  two  distinct  thresholds  of  performance  under  increasing 
latency.  The  first  is  the  level  of  latency  at  which  a  surgeon  can  first  detect  that  his  or  her  movements  are 
being  affected  by  the  communication  link.  Communication  latency  lower  than  this  level  is  consciously 
imperceptible  and  potentially  non-invasive  to  the  surgical  procedure.  Hence,  if  such  levels  can  be 
achieved  in  the  real  world,  then  telesurgery  may  be  safe  for  human  surgery  right  now.  The  second  level  is 
the  point  at  which  the  surgeon's  performance  is  degraded  to  the  point  that  the  surgery  cannot  be 
performed  safely  (Marescaux,  2002;  Lum,  2009).  This  level  is  identified  through  both  simulator  measured 
performance  and  the  expert  opinion  of  the  surgeon.  Between  the  first  and  second  thresholds,  a  surgeon 
may  be  able  to  successfully  manage  the  effects  of  latency  and  perform  a  safe  and  successful  procedure. 
Beyond  the  second  threshold,  telesurgery  would  be  considered  unsafe  with  the  available  equipment 
(Figure  2). 


Figure  2.  Conceptual  Diagram  of  Communication  Latency  Thresholds. 
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Figure  3.  Simulated  Surgical  Skills  Tasks 


We  further  hypothesize  that  more  experienced  surgeons  will  be  more  successful  at  managing  the  effects 
of  latency  and  would  be  the  best  practitioners  for  this  extension  of  robotic  surgery.  If  this  hypothesis  is 
correct,  then  surgeons  with  more  experience  should  achieve  higher  scores  and  shorter  completion  times  in 
the  simulation  experiment  that  we  are  performing. 

In  this  experiment,  subjects  performed  the  four  simulated  surgical  skills  exercises  shown  in  Figure  3. 
These  represent  many  of  the  core  skills  that  are  required  in  robotic  surgery.  Each  subject  performed  each 
exercise  three  times  as  described  above. 

Data  Collection 

Experimental  data  was  collected  by  the  simulator  software  and  manually  via  questionnaires.  Research 
proctors  administered  a  Pre-Test  questionnaire  on  the  level  of  surgical  experience  and  related  activities  of 
the  subject.  All  personal  and  performance  data  was  anonymized  to  insure  that  the  identity  of  the  subject 
could  not  be  linked  to  the  data  that  was  collected.  The  proctors  also  administered  a  Post -Test 
questionnaire  at  the  conclusion  of  each  of  the  skills  exercises  during  the  final  performance  stage.  The 
simulator  software  automatically  collected  multiple  measures  of  the  subjecf's  performance.  This  provided 
data  for  all  subjects  at  zero  latency,  during  their  familiarization  stage  with  latency,  and  during  the  final 
stage  which  is  the  focus  of  the  analysis.  This  data  will  allow  us  to  perform  multiple  analyses  of  the  skills 
of  robotic  surgeons  both  with  and  without  communication  latency. 

Pre-Test  Questionnaire 

The  Pre-Test  questionnaire  identified  multiple  items  of  demographic,  experience,  and  practice  data  on  the 
subjects.  These  included:  age,  gender,  dominant  hand,  surgical  status,  years  of  surgical  experience,  years 
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of  laparoscopic  experience,  years  of  robotic  experience,  number  of  weekly  procedures  in  laparoscopy  and 
robotics,  and  experience  with  laparoscopic  and  robotic  simulators,  as  well  as  with  video  games  and 
musical  instruments.  Additional  questions  captured  their  opinion  on  the  use  of  simulation  in  surgical 
education  and  certification. 

This  data  was  then  matched  to  the  data  from  their  performance  in  the  simulator. 

Simulator  Performance 

During  the  experiment,  the  simulator  itself  collected  a  number  of  data  points  on  each  subjecf's 
performance.  These  included:  time  to  complete,  overall  score,  total  hand  motion  in  centimeters,  master 
working  space,  number  of  instrument  collisions,  number  of  items  dropped,  excessive  instrument  force, 
distance  instruments  out  of  view,  incorrect  use  of  electrical  energy,  simulated  blood  loss,  and  number  of 
broken  blood  vessels. 

Post-Test  Questionnaire 

As  the  subjects  completed  their  final  repetition  of  each  of  the  four  skills  exercises,  the  proctor 
administered  a  post-test  questionnaire  which  asked  the  subject  for  their  opinion  on  the  stress  induced  by 
the  simulation  with  latency.  This  included  measures  of  the  mental  and  physical  demands  of  the  task,  the 
pace  of  the  task,  their  opinion  on  their  level  of  success,  the  amount  of  effort  expended,  the  level  of  mental 
discouragement  experienced,  and  their  perceived  complexity  of  the  exercise. 


Results 

The  analysis  of  data  from  the  first  54  subjects  is  provided  here.  Of  the  54  subjects  who  began  the 
experiment,  several  were  unable  to  complete  all  of  the  tasks  due  to  the  limited  amount  of  time  that  they 
could  devote  to  the  experiment.  Others  found  the  experiment  too  taxing  and  elected  to  terminate  their 
participation  before  completion.  As  a  result,  we  collected  complete  data  sets  without  latency  on  42 
subjects  and  complete  data  with  latency  on  only  21  of  those  subjects. 

This  data  was  analyzed  to  determine  the  level  of  correlation  between  the  subjects"  experience  and  their 
performance  both  with  and  without  latency.  For  the  non-latency  sample  size  of  42  and  alpha=0.05,  the 
Pearson  Product  Moment  Correlation  (PPMC)  value  is  0.304.  This  means  that  for  a  correlation  coefficient 
of  two  variables  in  this  size  of  sample  to  be  significant,  it  must  be  larger  than  the  PPMC  value. 

Table  1.  Correlation  Coefficients  without  Latency 


Exercise 

Overall 

Score 

Time  to 
Complete 

Pegboard  1 

0.141 

-0.110 

Camera  Targeting 

0.201 

-0.173 

Thread  the  Rings 

0.156 

-0.225 

Energy  Dissection 

0.267 

-0.217 

In  an  environment  without  any  latency  imposed  we  found  a  positive  correlation  between  years  of  robotic 
experience  and  overall  performance  score,  as  well  as  a  negative  correlation  between  experience  and  the 
total  time  to  complete  the  exercise  (Table  1).  Both  of  these  indicate  that  more  experience  leads  to  better 
performance  in  the  simulator  without  latency.  Though  this  correlation  is  consistently  supportive  that 
surgeons  with  more  experience  perform  non-latency  exercises  better  than  those  with  less  experience,  the 
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degree  of  this  correlation  is  not  large  enough  to  be  statistically  significant  for  this  sample  size.  The  data 
for  the  pegboard  exercise  is  graphed  in  Figures  4  and  5. 
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Figure  4.  Correlation  between  Robotic  Experience  and  Overall  Score  for  the  Peg  Board  exercise 
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Figure  5.  Correlation  between  Robotic  Experience  and  Time  to  Complete  for  the  Peg  Board 

exercise  without  communication  latency. 


When  latency  is  added,  the  subjects  with  similar  treatments  are  in  much  smaller  numbers.  For  example,  at 
a  latency  of  400ms  we  have  4  or  5  data  points  for  each  exercise  as  shown  in  Table  2.  The  analysis  of  this 
data  with  alpha=0.05  requires  a  PPMC  of  0.811  or  0.754  (respectively)  to  achieve  significance.  For  this 
small  number  of  data  points,  the  level  of  experience  does  not  contribute  to  the  ability  of  the  surgeon  to 
perform  better  under  latency.  The  data  for  the  pegboard  exercise  at  various  latency  levels  is  graphed  in 
Figures  6  and  7,  illustrating  the  lack  of  a  trend  for  both  surgeon  experience  and  latency  levels.  However, 
the  number  of  data  points  at  each  latency  level  is  still  so  small  that  definitive  conclusions  cannot  be  made 
about  the  issue. 


Table  2.  Correlation  Coefficients  with  400ms  Latency 


Exercise 

n 

Overall 

Score 

Time  to 
Complete 

Pegboard  1 

5 

0.093 

0.187 

Camera  Targeting 

4 

-0.915 

0.583 

Thread  the  Rings 

4 

-0.330 

0.262 
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Figure  6.  Correlation  between  Robotic  Experience  and  Overall  Score  for  the  Peg  Board  exercise 

with  various  communication  latencies. 


The  data  suggests  that  surgeons  who  have  more  experience  in  robotic  surgery  are  not  better  equipped  to 
self-manage  the  challenges  presented  by  communication  latency  in  telesurgery.  Subjects  with  little 
experience  are  as  likely  to  successfully  manage  latency  as  are  surgeons  with  more  experience. 

This  same  trend  holds  when  comparing  independent  variables  like  total  surgical  experience  and 
laparoscopic  experience  to  the  scores  achieved  in  the  simulator  with  latency. 

Conclusions 

The  lack  of  correlation  between  experience  and  telesurgical  performance  under  latency  refutes  our 
original  hypothesis  that  a  more  experienced  surgeon  would  more  successfully  manage  the  effects  of 
latency.  This  negative  finding  has  led  to  speculation  on  the  cause  of  these  results.  Several  may  be 
possible,  but  each  will  require  additional  experimentation.  First,  experienced  surgeons  may  be  very 
talented,  but  fixed,  in  their  methods  of  performing  surgery.  This  may  lead  them  to  perform  poorly  under 
latency  because  it  is  difficult  for  them  to  modify  their  behaviors,  where  inexperienced  surgeons  are  less 
ingrained  and  more  adaptable  to  the  situation.  Second,  since  the  simulator  is  a  computer-generated  virtual 
environment,  it  is  possible  that  surgeons  who  have  more  experience  in  simulators,  virtual  worlds,  and 
computer  games  may  have  developed  a  proficiency  for  solving  problems  in  this  kind  of  environment. 
They  may  also  have  experienced  latency  in  those  environments  and  developed  techniques  for 
compensating  for  it.  Third,  the  ability  to  manage  latency  may  be  related  to  the  physical  and  biological 
wiring  of  an  individual.  This  could  be  a  similar  phenomenon  to  the  tendency  for  some  people  to 
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experience  simulator  sickness,  while  others  do  not  suffer  from  it.  These  speculations  are  worthy  of  further 
investigation. 

Our  experience  with  subjects  has  been  that  latency  levels  above  500ms  severely  degrades  the  performance 
of  all  subjects.  In  many  cases  they  become  so  frustrated  or  exhausted  that  they  terminate  their 
participation  before  completion.  This  behavior  has  resulted  in  a  loss  of  a  number  of  valuable  data  points 
and  it  detracts  from  the  number  of  subjects  who  can  be  assigned  to  latency  levels  at  or  below  500ms 
where  we  have  found  the  best  results.  As  a  result,  we  have  modified  the  experiment  to  eliminate  the  use 
of  latency  levels  over  500ms.  As  we  continue  the  experiment  we  will  be  able  to  collect  more  data  at  lower 
latency  levels  where  the  most  interesting  and  useful  results  seem  to  reside.  This  modification  will  require 
an  adjustment  to  our  statistical  analysis  in  order  to  accurately  represent  results  across  this  change. 

The  objective  of  this  analysis  was  to  identify  the  degree  to  which  a  surgeon  can  compensate  for  the  effects 
of  latency  that  are  present  in  a  telesurgery  environment.  The  long-term  goal  is  to  identify  the  thresholds 
where  safe  and  successful  surgery  can  be  performed.  Our  findings  at  this  point  refute  our  hypothesis  that 
more  experienced  surgeons  would  be  able  to  manage  latency  more  successfully.  In  the  data  collected 
there  is  no  correlation  between  robotic  experience  and  the  ability  to  achieve  a  higher  score  in  the 
simulator  when  latency  is  inserted  into  the  procedure. 
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Robotic  Surgery:  Laparoscopic  Experience  and  Robotic  Performance 

There  is  some  question  as  to  the  degree  to  which  experience  in  laparoscopy  supports  superior 
performance  in  robotic  surgery.  Using  the  data  collected  in  the  telesurgery  experiment  we  examined  the 
relationship  between  experience  in  laparoscopy  and  performance  on  skills  exercises  in  a  robotic  simulator 
with  the  goal  of  determining  whether  the  correlation  is  positive  or  negative. 

Methods 

Surgeons  were  tested  in  their  ability  to  perform  four  different  simulated  robotic  surgical  skills  using  the 
dV-Trainer  simulators  (Mimic  Technologies,  Inc.,  Seattle,  WA)  of  the  da  Vinci  surgical  robot  (Intuitive 
Surgical  Inc.,  Sunnyvale,  CA).  The  subjects  completed  a  pre-test  questionnaire  to  provide  demographic 
and  experience  data,  which  included  the  number  of  years  of  practice  in  both  laparoscopic  and  robotic 
surgery.  The  simulator  collected  multiple  performance  metrics  during  each  of  the  four  exercises. 
Pearson"s  correlation  was  computed  on  the  relationship  between  the  number  of  years  of  laparoscopic  and 
robotic  experience,  and  their  overall  proficiency  score,  with  control  for  the  correlation  between  lap  and 
robotic  experience. 

Results 

A  total  of  54  subjects  participated  in  the  experiment  and  42  completed  all  four  experimental  tasks.  These 
subjects  reported  a  range  of  experience  in  laparoscopic  surgery  between  4  and  34  years,  and  in  robotic 
surgery  between  0  and  1 1  years.  For  this  analysis  those  indicating  zero  years  of  robotic  experience  were 
omitted,  reducing  the  sample  size  to  30  surgeons.  Using  a  Pearson  Correlation  (df=28,  alpha=0.05,  and 
significance  threshold  of  0.349)  we  found  a  statistically  significant  negative  correlation  between  years  of 
laparoscopic  experience  and  the  overall  proficiency  score  in  two  of  the  four  robotic  surgery  exercises 
(pegboard  =  -0.361;  thread  rings  =  -0.454),  and  a  negative  correlation  which  did  not  achieve  statistical 
significance  in  the  two  remaining  exercises  (peg  board  =  -0.152;  energy  dissection  =  -0.228).  This  data 
refuted  our  initial  assumption  that  more  years  of  laparoscopic  experience  would  indicate  higher  levels  of 
proficiency  in  robotic  skills.  The  data  shows  a  consistent  negative  correlation  between  these  variables.  We 
checked  for  a  possible  negative  correlation  between  the  number  of  years  of  laparoscopic  and  robotic 
surgical  experience,  which  would  indicate  that  surgeons  with  more  laparoscopic  experience  consistently 
have  less  robotic  experience  and  vice  versa.  However,  the  correlation  between  those  two  variables  was 
0.067,  indicating  almost  no  correlation  between  the  two. 

Conclusions 

Using  a  simulator  to  measure  the  proficiency  of  surgeons  with  both  laparoscopic  and  robotic  surgical 
experience  we  found  a  negative  correlation  between  the  number  of  years  of  laparoscopic  experience  and 
proficiency  the  exercises.  Surgeons  with  more  experience  in  laparoscopy  performed  worse  on  exercises  in 
a  robotic  simulator  device  than  those  with  less  experience  in  laparoscopy.  This  analysis  suggests  that 
years  of  laparoscopic  experience  may  be  detrimental  to  the  development  of  expertise  in  robotically- 
assisted  MIS. 


This  study  was  based  on  performance  with  a  dV-Trainer  (Mimic  Technologies,  Seattle,  WA)  simulator  of 
the  da  Vinci  robot  (Intuitive  Surgical,  San  Jose,  CA).  It  is  possible  that  the  differences  between  the 
simulator  and  the  real  robot  could  contribute  to  poor  performance  with  the  device  which  would  not  be 
present  if  we  were  using  the  actual  robot.  Analysis  of  our  data  showed  a  positive  correlation  between  the 
surgeons"  years  of  robotic  surgery  and  their  overall  scores  with  the  simulator,  suggesting  that  direct 
experience  with  robotics  does  contribute  to  better  performance  with  the  simulator  device.  Though 
positive,  this  correlation  did  not  reach  the  threshold  for  significance  (0.456)  for  this  sample 
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Surgical  Rehearsal 

Surgical  rehearsal  is  a  relatively  new  concept  in  the  medical  literature.  The  aim  of  its  introduction  to  the 
surgical  field  is  to  improve  surgical  performance,  outcomes  and  ensure  patient  safety.  However,  it  is 
imperative  to  differentiate  between  “simulation”  in  general  and  surgical  rehearsal  since  the  later  is  a 
“patient-specific  simulation”  which  gives  the  opportunity  to  rehearse  the  procedure  in  a  simulated 
environment,  using  the  real  patient's  data,  prior  to  performing  the  intervention  on  the  patient  rather  than 
exercising  general  surgical  tasks,  which  is  mere  simulation.  These  two  training  modalities  can  utilize  a 
simulator. 

A  number  of  simulators  have  been  developed  to  support  training  and  skill  assessment  in  robotic  surgery. 
The  currently  available  simulators  include:  the  Skills  Simulator  by  Intuitive  Surgical  Inc.,  aka  the 
“Backpack  Simulator”,  the  dV-Trainer  from  Mimic  Technologies  Inc.,  and  the  RoSS  by  Surgical  Sciences 
Inc.  All  of  these  simulators  utilize  a  visual  scene  that  is  presented  in  a  computer  generated  3D 
environment  that  represents  challenging  tests  of  dexterity  and  machine  operations.  In  the  first  simulator, 
the  trainee  sits  at  and  operates  a  console  that  is  identical  to  the  console  in  the  da  Vinci  surgical  system, 
just  as  if  she  or  he  were  doing  a  surgery.  Whereas  the  second  utilizes  a  desktop  device  that  replicates  the 
hardware  of  the  da  Vinci  surgical  system  surgeon"s  console,  it  includes  a  3D  simulated  environment  that 
is  identical  to  the  former  software.  The  third  is  similar  in  that  it  uses  a  simulated  hardware  device  in  place 
of  the  real  robotic  equipment. 

“Personalization”  of  the  surgical  procedure  implies  the  conversion  of  a  digital  CT  scans  (or  other 
radiological  images)  into  a  Virtual  Reality  (VR)  system  that  is  interactive  within  the  simulator  system  to 
the  surgeon"s  intervention.  This  system  creates  a  3D  reconstruction  of  the  organ  and  the  surgeon  can 
rehearse  a  single  step  or  a  complete  surgery  (Anvari,  2004).  Unfortunately,  this  is  not  currently  available 
in  the  robotic  simulators  and  is  considered  a  rate-limiting  step  toward  any  effort  to  study  robotic  surgical 
rehearsal  (Satava  &  Simon,  1993).  However,  this  type  of  simulation  has  been  performed  in  laparoscopic 
surgery  (Willaert  et  al,  2010)  and  may  be  available  for  use  in  robotic  simulation  in  the  near  future. 

[Note:  The  following  experimen  t  is  still  in  draft  protocol  developmen  t  and  has  not  yet  been  approved  by 
the  government  sponsor.] 

Given  the  current  state  of  simulator  technology,  it  is  possible  to  perform  near  identical  procedures  only 
when  using  skills  devices  with  the  robot  which  already  exist  within  the  virtual  models  of  the  simulators. 
For  the  purposes  of  this  research,  we  will  use  a  simulator,  a  skills  device,  and  a  porcine  model.  The  dV- 
Trainer  contains  a  number  of  exercises,  two  of  which  can  be  leveraged  for  this  study.  The  simplest 
experiment  will  compare  performance  on  an  existing  skills  exercise  called  the  “Matchboard”.  Subjects 
will  perform  this  exercise  in  the  simulator  for  rehearsal.  After  a  given  number  of  repetitions,  or  upon 
achieving  proficiency  (protocol  is  still  under  design),  they  will  move  to  the  actual  da  Vinci  robot  and 
perform  this  identical  skill  using  a  Matchboard  object  that  has  been  manufactured  to  the  exact  dimensions 
and  specifications  of  the  VR  object  in  the  simulator  (Figure  8).  The  performance  of  this  group  will  be 
compared  to  a  control  group  who  receives  instructions  on  performing  the  exercise  on  the  robot,  but  who 
will  attempt  it  without  experience  in  the  simulator. 
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Figure  8.  Dry  Lab  Surgical  Rehearsal  Experiment 


A  more  complex  version  of  this  experiment  has  been  designed  which  compares  performance  on  the 
simulator  with  performance  on  a  live  porcine  model.  The  current  simulators  contain  a  limited  number  of 
exercises  that  approximate  live  tissue  operations.  The  companies  that  make  these  devices  have  stated  that 
they  are  not  presently  prepared  to  create  a  more  animate  exercise  in  their  simulators.  Therefore,  we  have 
identified  an  Energy  Dissection  exercise  in  the  dV -Trainer  for  which  there  is  a  rough  equivalent  in  a 
porcine  model  (Figure  9).  The  experimental  group  will  rehearse  the  exercise  in  the  simulator  and  then 
perform  an  approximately  equal  procedure  in  the  porcine  model.  Their  performance  will  be  compared  to 
that  of  a  control  group  who  will  operate  on  the  porcine  model  after  instruction  on  the  procedure,  but 
without  experience  on  the  simulator. 


Figure  9.  Wet  Lab  Surgical  Rehearsal  Experiment 


The  data  from  both  of  these  experiments  will  be  analyzed  to  measure  the  difference  in  performance  for 
subjects  who  receive  a  simulation  rehearsal  prior  to  performing  a  nearly  identical  task  with  the  real  robot. 
This  data  will  inform  the  community  on  the  potential  for  using  simulators  as  pre -operative  planning  and 
rehearsal  systems.  Other  researchers  have  conducted  experiments  on  the  effects  of  generic  warm-up 
exercises  with  the  simulator.  Surgical  rehearsal  extends  that  concept  to  a  procedure  specific  activity 
which  will  eventually  become  a  realistic  representation  on  surgery  on  CT  images  of  the  patient' 's  personal 
anatomy. 
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Robotic  Curriculum 

We  are  collaborating  with  a  grant  to  the  Minimally  Invasive  Robotics  Association  (MIRA)  to  create  a 
fundamental  curriculum  in  the  field  of  robotic  surgery.  Following  the  process  and  lessons  learned  from 
the  creation  of  the  Fundamentals  of  Laparoscopic  Surgery  (FRS),  this  joint  project  is  referred  to  as  the 
Fundamentals  of  Robotic  Surgery  (FRS). 

Methods 

This  project  has  resulted  in  three  meetings  of  leading  robotic  surgeons  from  around  the  world.  As  a  group 
they  have  agreed  upon  a  set  of  outcomes  measures  and  a  draft  curriculum.  They  have  also  designed  a 
multi-skills  testing  device  for  use  in  the  psychomotor  skills  portion  of  the  curriculum. 

We  have  created  a  process  and  a  group  of  participants  to  unify  the  previous  attempts  to  develop  a  robotic 
curriculum  and  expand  to  a  much  larger  foundation  of  surgical  societies  with  a  stake  in  this  new 
technology. 

Participation  in  this  effort  was  invited  from  multiple  certifying  boards,  professional  surgical  societies,  and 
associations  that  represent  international  practitioners  and  regulators  of  various  surgical  specialties  as  well 
as  the  United  States  Department  of  Defense  (DoD)  and  Veterans  Flealth  Administration  (VF1A)  (Table  3). 
The  conference  participants  are  members  of  these  organizations  or  agencies  and  are  selected  to  be  able  to 
provide  insight  into  the  needs  of  their  organizations,  but  they  do  not  represent  an  endorsement  or 
acceptance  of  the  results,  and  participation  does  not  imply  acceptance  by  the  societies,  boards  or  agencies. 
However,  the  AUA,  AAGL,  and  SAGES  elected  to  appoint  and  send  representatives  who  could  officially 
speak  for  their  organizations"  needs  for  a  robotic  curriculum  and  officially  accept  the  results  of  the 
consensus  conferences.  This  project  is  an  effort  to  provide  the  stakeholders  with  the  best  scientific 
evidence  upon  which  to  base  their  decisions  regarding  implementation  of  a  fundamental  curriculum  to 
meet  their  needs  while  reducing  redundancy,  competition  and  duplication  of  effort. 

Table  3.  Invited  Organizational  Representation  in  Fundamentals  of  Robotic  Surgery. 

American  Association  Gynecologic  Laparoscopy  (AAGL)  * 

American  College  of  Surgeons  (ACS) 

American  Congress  of  Obstetrics  and-Gynecology  (ACOG) 

American  Urologic  Association  (AUA)  * 

American  Academy  of  Orthopedic  Surgeons  (AAOA) 

American  Association  of  Thoracic  Surgeons  (AATS) 

American  Association  of  Colo-rectal  Surgeons  (ASCRS) 

Minimally  Invasive  Robotic  Association  (MIRA) ' 

Society  for  Robotic  Surgery  (SRS) 

Society  of  American  Gastrointestinal  and  Endoscopic  Surgeons  (SAGES)  * 

American  Board  of  Surgery  (ABS) 

Accreditation  Council  of  Graduate  Medical  Education  (ACGME) 

Association  of  Surgical  Educators  ( ASE) 

Residency  Review  Committee  (RRC)  -  Surgery 
Royal  College  of  Surgeons-lreland  (RCS1) 

Royal  College  of  Surgeons-London  (RCSL) 

Royal  College  of  Surgeons-Australia  (RCSA) 

U.S.  Department  of  Defense  (DoD)  r 

U.S.  Department  of  Veterans  Health  Affairs  (VHA) 

*  :  Official  Representative  Participation 
1  Funding  organizations. 
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Each  consensus  conference  was  conducted  over  a  two-day  period  using  a  modified  Delphi  Method 
(Dalkey,  1969).  This  methodology  consisted  of  a  facilitator  who  captured  the  input  and  guidance  of  the 
participants.  This  input  was  then  analyzed  for  common  concepts  to  create  a  list  of  critical  items  in  robotic 
surgery.  Previously  published  material  from  a  single  institution"s  curriculum  was  used  as  a  template  for 
initial  idea  generation  (Dulan  et  al,  2012a,  2012b).  The  individual  outcomes  measures  and  curriculum 
materials  were  itemized  and  votes  taken  on  their  importance  according  to  each  participant.  This  method 
led  to  a  composite  ranking  which  was  captured  in  a  draft  report.  The  report  containing  the  first  group 
ratings  was  then  sent  to  each  participant  for  their  private  deliberation.  Each  participant  then  submitted  a 
second  set  of  scores  which  were  informed  by  the  first  composite  scores,  but  anonymous  to  other  group 
members.  This  modified  Delphi  Method  led  to  a  higher  level  of  consensus  around  the  measures  and  the 
curriculum.  It  also  identified  those  items  for  which  there  was  little  group  support.  Those  items  were 
removed  from  the  list  of  outcomes  measures  and  from  the  outline  of  the  curriculum. 

The  first  conference  on  outcomes  measures  was  attended  by  20  participants  that  included  surgeons, 
scientists,  educators,  and  facilitators.  The  ranking  of  the  tasks  identified  was  done  by  a  subset  of  nine 
experienced  surgeons.  Participants  who  were  not  surgeons  abstained  from  the  scoring  process. 

The  second  conference  on  curriculum  development  was  attended  by  38  surgeons,  scientists,  educators, 
and  facilitators.  This  group  reviewed  and  became  familiar  with  the  material  from  the  first  conference. 
Thereupon,  they  were  divided  into  three  working  groups  to  develop  curriculum  that  focused  on  didactic 
and  knowledge-based  information,  psychomotor  skills,  and  team  training  and  communications.  Similarly, 
the  actual  ranking  of  the  material  developed  was  limited  to  experienced  surgeons  within  the  group. 

Results 

The  first  consensus  conference  resulted  in  a  list  of  25  outcomes  measures  which  the  group  agreed  should 
be  mastered  by  a  surgeon  seeking  privileges  in  robotics.  These  included  8  pre-operative,  15  intra¬ 
operative  and  2  post-operative  tasks  which  are  shown  in  Table  4.  The  resulting  report  also  provides 
detailed  definitions,  descriptions,  errors,  outcomes  and  metrics  for  each  of  these  tasks  (Satava  et  al,  2012). 


18 


Annual  Report,  September  2012. 

Project  W81XWH-1 1-2-0158  Medical  Robotic  and  Telesurgical  Simulation  and  Education  Research 


Table  4.  FRS  Ou  tcomes  Measures  from  Consensus  Conference 


Pre-Operative 


System  Settings 


Intra-Operative 


Energy  Sources 


Post-Operative 


Transition  to  Bedside  Asst 


Ergonomic  Positioning 
Docking 
Robotic  Trocars 
OR  Set-up 

Situation  Awareness 
Closed  Loop  Comms 

Respond  to  System  Errors 


Camera  Control 
Clutching 

Instrument  Exchange 
Foreign  Body  Management 

Multi-arm  Control 
Eye-hand  Instrument  Coord 

Wrist  Articulation 


Undocking 


Atraumatic  Tissue  Handling 


Dissection  -  Fine  &  Blunt 


Cutting 

Needle  Driving 


Suture  Handling 
Knot  Tying 


Safety  of  Operative  Field 


The  second  consensus  conference  on  curriculum  development  resulted  in  outlines  and  principles  for  the 
creation  of  a  curriculum  to  teach  the  previously  identified  list  of  tasks  and  knowledge. 

Didactic  and  Knowledge.  The  didactic  and  knowledge  working  group  created  an  outline  of  the  material 
which  should  be  taught  in  lecture  format.  This  will  include: 

1.  Introduction  to  robotic  surgical  systems. 

2.  Pre-operative  set-up  of  equipment  and  positioning  of  staff. 
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3.  Intra-operative  use  of  a  robot,  surgeon  ergonomics,  visual  field  control,  and  necessary 
instruments  and  supplies. 

4.  Post-operative  steps  for  removing  a  robot  and  transitioning  to  bedside  control. 

Each  of  these  included  an  explicit  list  of  errors  that  can  occur  in  the  process. 

Psychomotor.  The  psychomotor  skills  working  group  prefaced  their  work  with  seven  principles  that 
should  be  applied  in  selecting  or  designing  a  skills  device  for  robotic  surgery.  Those  principles  were: 

1 .  The  tasks  should  be  3  dimensional  in  nature. 

2.  The  tasks  designed  for  testing  should  be  such  that  they  have  multiple  learning  objectives  that 
incorporate  multiple  tasks  from  the  first  conference  report.  The  tasks  designed  for  training  will 
have  more  focused  learning  objectives. 

3.  Implementation  of  the  tasks  and  the  resultant  method  for  teaching  should  be  cost  effective. 

4.  High  fidelity  models  should  be  used  for  testing.  Training  can  use  lower  fidelity  devices  or 
methods. 

5.  Tasks  should  be  easy  to  administer  to  ensure  Inter-Rater  Reliability  (IRR). 

6.  The  tasks  should  be  designed  for  implementation  with  physical  objects  and  devices.  Future 
implementation  in  VR  with  a  simulator  would  be  derivative  of  the  physical  model. 

7.  Preference  should  be  given  to  tasks  that  have  existing  evidence  of  validity 

The  group  then  identified  16  of  the  25  tasks  which  contained  psychomotor  features.  To  address  these, 
they  proposed  ten  tasks  which  could  be  used  to  measure  these  skills.  Three  tasks  were  drawn  from  FLS, 
others  were  selected  from  existing  educational  programs,  and  designs  for  new  task  devices  were 
proposed. 

1 .  FFS  peg  transfer 

2.  FFS  suturing 

3.  FFS  pattern  cutting 

4.  Running  Suture 

5.  Dome  with  four  towers 

6.  Vessel  dissection  and  clipping 

7.  UTSW  4th  arm  retraction  and  cutting 

8.  Energy  and  mechanical  cutting 

9.  Docking  task  (new  design) 

10.  Trocar  insertion  task  (new  design) 

For  each  of  these  the  group  also  identified  the  associated  task  description,  conditions,  metrics,  and  errors. 
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Figure  10:  Design  for  FRS  Psychomotor  Skills  Testing  Device 


Team  Training  and  Communications.  The  team  training  and  communications  working  group  prefaced 
their  work  by  defining  the  importance  of  team  training  in  a  robotic  environment.  They  identified  the 
following  principles  as  essential  to  successful  team-based  operations  and  training. 

1 .  Inclusion 

2.  Empowerment 

3.  Person  specific 

4.  Reiterative 

5.  ,  Just  in  time" 

6.  Ownership 

7.  Risk  management/quality  improvement-  closed  loop 

They  stated  that  existing  programs  like  TeamSTEPPS  can  be  applied  to  robotic  teams.  Their  curriculum 
follows  a  checklist  format  and  is  conceptually  derived  from  the  standard  WEIO  checklist.  For  robotic 
training  they  recommended  the  following  checklists: 

1.  Pre-operative.  Addressing  General  situation,  surgeon,  anesthetist,  nurse/OPD,  and  surgical  site 
infection. 

2.  Robotic  Docking.  Addressing  anesthesia,  patient,  bedside  assist,  procedure-specific  checks,  and 
trouble  shooting. 

3.  Intra-operative.  Addressing  the  communication  that  occurs  within  a  team  throughout  the 
operation. 

4.  Undocking  and  Debriefing. 

A  third  consensus  conference  was  held  in  August  2012  to  write  the  detailed  material  that  to  be  included  in 
the  didactic  and  team  training  sections  of  the  curriculum;  and  a  specific  psychomotor  skills  device  was 
designed. 

Conclusions  &  Discussion 

Three  consensus  conferences  involving  members  from  major  stakeholder  organizations  in  surgical 
training,  governance,  and  certification  across  multiple  specialties  have  been  conducted  to  arrive  at  a 
consensus  regarding  the  most  important  outcome  measures  for  the  safe  conduct  of  robotic  surgery  and  the 
curriculum  to  teach  those  skills  and  knowledge.  The  development  of  FRS  is  multi-specialty,  system 
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agnostic  and  follows  decades  of  experience  in  other  industries  at  developing  such  education  and  training 
platforms.  Using  the  curriculum  for  training  and  assessment  should  result  in  a  surgeon  who  has 
proficiency  in  basic  robotic  surgery  skills  and  is  capable  of  passing  the  requirements  of  high  stakes  testing 
and  evaluation.  At  some  future  time,  this  testing  and  evaluation  would  be  administered  by  an  appropriate 
independent,  objective  and  authoritative  organization,  which  would  adopt  the  materials  developed 
through  this  consensus  process. 

Participants 

In  addition  to  the  two  funding  agencies,  this  project  is  a  collaboration  of  leading  robotic  surgeons  and 
educators.  The  following  have  all  participated  in  and  contributed  to  the  creation  of  the  materials  reported 
here: 

A.  Advincula;  R.  Aggarwal;  A.  A1  Ansari;  D.  Albala;  R.  Angelo;  M.  Anvari;  J.  Armstrong;  G. 

Ballantyne;  M.  Billia;  J.  Borin;  D.  Bouchier-Hayes;  T.  Brand;  S.  Chauhan;  P.  Coelho;  A.  Cuschieri;  B. 
Dunkin;  S.  Dunlow;  V.  Ficarra;  A.  Gallagher;  L.  Glazerman;  T.  Grantcharov;  D.  Hananel;  J.  Hebert;  R. 
Holloway;  W.  Judd;  K.  Kim;  M.  Koch;  T.  Kowalewski;  R.  Kumar;  K.  Kunkler;  G.  Lee;  T.  Lendvay;  R. 
Leveillee;  J.  Levy;  G.  Maddem;  S.  Magnuson;  M.  Marohn;  D.  Maron;  M.  Martino;  P.  Neary;  K.  Palmer; 
E.  Parra-Davila;  V.  Patel;  S.  Ramamoorthy;  K.  Rha;  J.  Riess;  B.  Rocco;  R.  Rush;  R.  Satava;  D.  Scott;  N. 
Seymour;  M.  Sinanan;  R.  Smith;  D.  Stefanidis;  C.  Sundaram;  R.  Sweet;  E.  Verrier;  G.  Weinstein 
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Key  Research  Accomplishments 

•  Telesurgery:  Communications  Latency.  Performance  under  latency  is  not  correlated  with  years  of 
experience  in  robotics  or  laparoscopy. 

•  Robotic  Surgery  Skills.  Extensive  years  of  laparoscopic  experience  are  detrimental  to  acquiring 
robotic  surgery  skills. 

•  Robotic  Curriculum:  Consensus  Conferences.  Outcomes  measures  and  a  curriculum  for  certifying 
robotic  surgeons  have  been  developed.  Validation  will  be  performed  in  the  second  year  of  the 
grant. 

•  Simulation:  Surgical  Rehearsal.  A  protocol  for  exploring  the  effects  of  procedure-specific 
simulation  rehearsal  on  surgical  performance  is  being  developed. 
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Reportable  Outcomes 

Manuscripts 

Satava,  Chauhan,  Smith,  &  Patel.  “Fundamentals  of  Robotic  Surgery  Consensus  Conference  1:  Outcomes 
Measures”,  Surgery  (an  ACS  journal).  Submitted  April  2012. 

Smith  &  Chauhan.  “Using  Simulators  to  Measure  Communication  Latency  Effects  in  Robotic 
Telesurgery”,  2012  Interservice/Industry  Training  Education  and  Simulation  (I/ITSEC)  Conference . 
December  2012. 

Abstracts 

Satava,  Smith  &  Patel.  “Report  on  the  First  Consensus  Conference  on  the  Fundamentals  of  Robotic 
Surgery”  Outcomes  Measures”,  ACS  Accredited  Education  Institutes  Meeting.  March  2012. 

Presentations 

Satava  &  Smith.  “Fundamentals  of  Robotic  Surgery  (FRS):  Overview  and  Results  of  First  Two 
Consensus  Conferences”  Society  for  Laparoscopic  Surgeons  Annual  Meeting,  September  2012 

Smith.  “Robotic  Surgery  and  Surgical  Simulation”,  presentation  to  International  Council  on  Systems 
Engineering  -  Orlando  Chapter.  February  2012. 

Smith.  “Beyond  Education  and  Training:  Challenges  of  Running  Medical  Simulators  in  New  Paradigms”. 
International  Meeting  on  Simulation  in  Healthcare.  January  2012. 

Smith.  "Simulation  in  Surgical  Education",  presentation  to  American  College  of  Healthcare  Executives  - 
Orlando  Chapter,  December  2011. 

Smith.  "Medical  Simulation  Special  Event:  Robotic  and  Telesurgery  Research  Using  Simulation", 
Interservice/Industry  Training,  Simulation,  and  Education  Conference,  December  2011. 

Smith.  "Robotic  and  Telesurgery  Research",  presentation  to  National  Center  for  Simulation  -  Quarterly 
Meeting,  October  2011. 
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Conclusion 

Telesurgery:  Communications  Latency  Experiments.  As  of  August  30,  2012,  84  subjects  have  participated 
in  this  experiment.  We  continue  to  collect  subject  data.  The  data  collected  in  the  experiment  has  shed 
light  on  a  number  of  details  around  robotic  surgery.  In  general  we  find  that  the  effect  of  latency  on 
individual  surgeons  is  not  predictable  by  their  levels  of  experience  in  either  robotics  or  laparoscopy.  The 
performance  under  latency  varies  widely  at  all  latency  settings.  The  majority  of  subjects,  though  not  all, 
can  manage  latency  at  200ms  and  below.  Above  this,  most  subjects  of  all  experience  levels  exhibit  severe 
drops  in  performance.  Previous  research  projects  have  suggested  that  latency  below  250ms  could  be 
considered  manageable  because  it  is  below  the  perception  threshold  of  most  people.  Our  experiments 
agree  with  this  for  a  majority  of  subjects,  though  certainly  not  for  all. 

The  data  from  this  experiment  also  allowed  us  to  study  the  relationship  between  years  of  laparoscopic 
experience  and  performance  in  a  robotic  environment.  We  found  a  consistent  negative  correlation 
between  these  two  variables.  It  appears  that  extensive  laparoscopic  experience  is  detrimental  to  the 
acquisition  of  skills  in  robotic  surgery. 

Papers  on  both  of  these  results  have  been  submitted  for  presentation  at  conferences  and  are  being 
prepared  for  journal  publication. 

Simulation:  Military-use  Validation.  At  the  project  kick-off  of  this  grant  in  September,  2011  we  requested 
to  move  this  experiment  to  year  two  of  the  grant.  This  was  motivated  by  the  lead  time  necessary  to 
purchase  the  equipment  required.  This  change  was  accepted  by  the  government  and  this  project  will  be 
performed  in  the  second  year. 

Robotic  Curriculum:  Consensus  Conferences.  We  have  held  three  conferences  consisting  of  leading 
robotic  surgeons  from  around  the  world.  Eighteen  participated  in  the  outcomes  measures  meeting,  forty 
participated  in  the  curriculum  development  meeting,  and  twenty-four  participated  in  the  curriculum 
writing  meeting.  We  have  developed  a  list  of  25  outcomes  measures  that  robotic  surgeons  need  to  be  able 
to  demonstrate.  Three  different  curriculums  have  been  created  -  didactic,  psychomotor  skills,  and  team 
training.  A  multi-skills  device  for  testing  many  of  the  outcomes  has  been  designed  and  will  be  produced 
in  the  second  year. 

We  are  in  discussions  with  SAGES  to  develop  a  High  Stakes  Test  for  this  material  and  to  create  a  system 
to  administer  that  test. 


Simulation:  Surgical  Rehearsal.  At  the  government  kick-off  meeting  we  requested  to  move  this 
experiment  to  year  one  in  place  of  the  military-use  validation  study.  Approval  for  that  change  was  given 
by  the  government.  Multiple  protocols  were  designed  for  conducting  this  experiment;  however,  none  of 
them  were  both  feasible  for  implementation  and  acceptable  to  the  government  reviewers.  A  new  protocol 
is  in  draft  now  and  will  be  presented  to  the  government  for  review  at  the  end  of  September  2012.  A 
summary  of  that  protocol  is  included  in  this  report,  but  has  not  been  completed  or  approved  at  this  time. 
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Appendices 

Copies  of  manuscripts,  abstracts,  and  presentations  of  work  resulting  from  this  grant  are  included  as 
appendices  in  separate  documents. 
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Contribution  of  laparoscopic  surgical  experience  to  the  development  of  robotic  proficiency 
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A.  Advincula,  MD 

Florida  Hospital,  Celebration  Health 
( other  affiliations  ? ) 
Celebration,  Florida 


Abstract 

Objectives:  There  is  some  question  as  to  the  degree  to  which  experience  in  laparoscopy  supports 
superior  performance  in  robotic  surgery.  This  study  examined  the  relationship  between 
experience  in  laparoscopy  and  performance  on  skills  exercises  in  a  robotic  simulator  with  the 
goal  of  determining  whether  the  correlation  is  positive  or  negative. 

Methods:  Surgeons  were  tested  in  their  ability  to  perform  four  different  simulated  robotic 
surgical  skills  using  the  dV-Trainer  simulators  ( Mimic  Technologies,  Inc.,  Seattle,  WA)  of  the  da 
Vinci  surgical  robot  (Intuitive  Surgical  Inc.,  Sunnyvale,  CA).  The  subjects  completed  a  pre-test 
questionnaire  to  provide  demographic  and  experience  data,  which  included  the  number  of  years 
of  practice  in  both  laparoscopic  and  robotic  surgery.  The  simulator  collected  multiple 
performance  metrics  during  each  of  the  four  exercises.  Pearson ’s  correlation  was  computed  on 
the  relationship  between  the  number  of  years  of  laparoscopic  and  robotic  experience,  and  their 
overall  proficiency  score,  with  control  for  the  correlation  between  lap  and  robotic  experience. 

Results:  A  total  of  54  subjects  participated  in  the  experiment  and  42  completed  all  four 
experimental  tasks.  These  subjects  reported  a  range  of  experience  in  laparoscopic  surgery 
between  4  and  34  years,  and  in  robotic  surgery  between  0  and  11  years.  For  this  analysis  those 
indicating  zero  years  of  robotic  experience  were  omitted,  reducing  the  sample  size  to  30 
surgeons.  Using  a  Pearson  Correlation  (df=28,  cilpha=0.05,  and  significance  threshold  of  0.349) 
we  found  a  statistically  significant  negative  correlation  between  years  of  laparoscopic  experience 
and  the  overall  proficiency  score  in  two  of  the  four  robotic  surgery  exercises  (pegboard  =  -0.361; 
thread  rings  -  -0.454),  and  a  negative  correlation  which  did  not  achieve  statistical  significance 
in  the  two  remaining  exercises  (peg  board  =  -0.152;  energy  dissection  =  -0.228).  This  data 
refuted  our  initial  assumption  that  more  years  of  laparoscopic  experience  would  indicate  higher- 
levels  of  proficiency  in  robotic  skills.  The  data  shows  a  consistent  negative  correlation  between 
these  variables.  We  checked  for  a  possible  negative  correlation  between  the  number  of  years  of 
laparoscopic  and  robotic  surgical  experience,  which  would  indicate  that  surgeons  with  more 
laparoscopic  experience  consistently  have  less  robotic  experience  and  vice  versa.  However,  the 
correlation  between  those  two  variables  was  0.067,  indicating  almost  no  correlation  between  the 
two. 

Conclusions:  Using  a  simulator  to  measure  the  proficiency  of  surgeons  with  both  laparoscopic 
and  robotic  surgical  experience  we  found  a  negative  correlation  between  the  number  of  years  of 
laparoscopic  experience  and  proficiency  the  exercises.  Surgeons  with  more  experience  in 
laparoscopy  performed  worse  on  exercises  in  a  robotic  simulator  device  than  those  with  less 
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experience  in  laparoscopy.  This  analysis  suggests  that  years  of  laparoscopic  experience  may  be 
detrimental  to  the  developmen  t  of  expertise  in  robotically-assisted  MIS. 

This  study  was  based  on  performance  with  a  dV-Trainer  ( Mimic  Technologies,  Seattle,  WA) 
simulator  of  the  da  Vinci  robot  (Intuitive  Surgical,  Scm  Jose,  CA).  It  is  possible  that  the 
differences  between  the  simulator  and  the  real  robot  could  contribute  to  poor  performance  with 
the  device  which  would  not  be  present  if  we  were  using  the  actual  robot.  Analysis  of  our  data 
showed  a  positive  correlation  between  the  surgeons  ’s  years  of  robotic  surgery  and  their  overall 
scores  with  the  simulator,  suggesting  that  direct  experience  with  robotics  does  contribute  to 
better  performance  with  the  simulator  device.  Though  positive,  this  correlation  did  not  reach  the 
threshold  for  significance  (0.456)  for  this  sample  size. 


References 

1 .  Intuitive  Surgical.  Investor  Presentation  Q3  2012.  Accessed  on  July  24,  2012  at  http://phx.corporate- 
ir.net/phoenix.zhtml?c=122359&p=irol-lRHome 

2.  "Concurrent  and  Predictive  Validation  of  a  Novel  Robotic  Surgery  Simulator:  A  Prospective, 
Randomized  Study",  Journal  of  Urology,  February  2012,  by  A.  Hung,  M.  Patil,  P.  Zehnder,  J  Cai,  C. 
Ng,  M.  Aron,  I.  Gill,  M.  Desai 

3.  "The  Virtual  Reality  Simulator  dV-Trainer®  is  a  Valid  Assessment  Tool  for  Robotic  Surgical  Skills", 
Surgical  Endoscopy,  April  2012,  by  C.  Perrenot,  M.  Perez,  N.  Tran,  J-P.  Jehl,  J.Felblinger 

4.  "Validation  Study  of  a  Virtual  Reality  Robotic  Simulator  -  Role  as  an  Assessment  Tool?",  Journal  of 
Urology,  March  2012,  by  J.  Lee,  D.  Kerbl,  P.  Mucksavage,  M.  Etafy,  V.  Huynh,  E.  McDougall 

5.  "Validating  the  Use  of  the  Mimic  dV-Trainer  for  Robotic  Surgery  Skill  Acquisition  Among  Urology 
Residents",  Urology,  October  2011,  by  R.  Korets,  A.C.  Mues,  J.A.  Graversen,  M.  Gupta,  M.C. 
Benson,  K.L.  Cooper,  J.  Landman,  K.K.  Badani 

6.  "Face,  Content  and  Construct  Validity  of  a  Novel  Robotic  Surgery  Simulator",  Journal  of  Urology, 
September  2011,  by  A.  Hung,  P.  Zehnder,  M.  Patil,  J  Cai,  C.  Ng,  M.  Aron,  1.  Gill,  M.  Desai 

7.  "Does  Training  on  a  Virtual  Reality  Robotic  Simulator  Improve  Performance  on  the  da  Vinci  ® 
Surgical  System?”,  Journal  of  Endourology,  Volume  24,  Number  3,  March  2010,  by  M.  Lemer,  M. 
Ayalew,  W.  Peine,,  C.  Sundaram 

8.  “Face,  Content,  and  Construct  Validity  of  dV-Trainer,  a  Novel  Virtual  Reality  Simulator  for  Robotic 
Surgery",  Journal  of  Urology,  Volume  73,  Issue  6,  June  2009,  by  P.  Kenney,  M.  Wszolek,  J.  Gould, 

J.  Libertino,  A.  Moinzadeh 

9.  “Initial  validation  of  a  virtual-reality  robotic  simulator”,  Journal  of  Robotic  Surgery,  Volume  2, 
Number  3,  September  2008,  by  T.  Lendvay,  P.  Casale,  R.  Sweet,  C.  Peters 


29 


FLORIDA  HOSPITAL 

NICHOLSON  CENTER 

FOR  SURGICAL  ADVANCEMENT 


Simulation  in  Surgical  Education: 

Innovation  to  reduce  training  time,  increase  case 
access,  increase  expertise,  and  reduce  errors 


Roger  Smith,  PhD,  DM,  MS,  MBA 
Chief  Technology  Officer 
Florida  Hospital 

Nicholson  Center  for  Surgical  Advancement 


roger.smith@flhosp.org 


t 


AmericanCollcgc  of 
Healthcajy  Executives 

for  Irnden  who  cart  ® 


Approved  for  Public  Release. 


Errors  Eliminate  Profits 


*  Minor  Complication 

-  Revisit  eliminates  all  profit  from  the  original  surgery 

*  Major  Complication 

-  Revisit  costs  3X  the  profit  from  the  original  surgery 
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\  Creating  Experts  &  Eliminating  Errors 


Outliers 

e 

The  Story  of  Success 

Malcolm 

Gladwell 


10,000  hours  to  become  an  expert  -  Gladwell 
“There  is  no  excuse  for  the  surgeon  to  learn  on  the  patient.”  -  William  Mayo,  1927 
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Medical  Education  -  Explosion  of  Information 


•  Medical  procedures  are  becoming  more  numerous  and  more 
complex  -  medical  knowledge  has  “hypertrophied”  (cooked 


•  Training  residents  to  a  common  level  of  knowledge  and 

competence  is  already  impossible  (Satava,  2008) 
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“The  Perfect  Storm” 


•  Risk  to  patient  health.  (McDougall, 

2007) 

•  Ethics  of  practicing  on 

patients.  (Satava,  2004;  Murphy,  2007) 

•  Cost  is  a  barrier  to  training. 

(Bridges,  1999) 

•  Insurance  coverage  of 
educational  actions.  <satava,  2004) 

•  Working  hour  limits.  (Satava,  2004) 

•  Availability  of  training 

opportunities.  (Birden,  2007;  Davis,  1999) 

•  Access  to  training.  (Dunkin,  2007;  Spitzer, 

1997) 


,  2007) 


•  Complexity  of  modern  surgery. 

(McDougall,  2007) 

•  Volume  of  unique  procedures. 

(Reznick  &  MacRae,  2006) 

•  Proficiency-based  Medicine. 

(Murray,  2005) 

•  Quality  of  technology.  (Murphy,  2007) 

•  Expectations  around  computer 

technologies.  (Murphy,  2007) 

•  Acceptance  of  technology.  <ziv, 

2003) 

•  Learning  from  Mistakes.  (Ziv,  2005) 


Objectives  for  Simulation  in  Education 


*  Objective  1:  Reduce  Cost 

*  Objective  2:  Increase  Case  Access 

*  Objective  3:  Reduce  Training  Time 

*  Objective  4:  Reduce  Errors 

Similar  Motivations  in  Military,  Industrial,  and  Medical  Training 


/A 


Objective  1 :  Reduced  Cc 


•  Surgery  as  a  teaching  event  consumes 
resources  that  could  generate  additional 

revenue.  (Bridges  &  Diamond  1999) 

-  1 86  hours  over  a  4  year  residency 

-  Estimate  OR  costs  at  $257.40  per  hour. 

-  Adds  $47,970  to  the  cost  of  a  medical 
education. 

•  Updated:  Adds  $186,363  to  $279,545 
during  four  year  residency 

-  US  OR  is  $1 ,500  per  hour  (Frost  &  Sullivan,  2004) 

-  Swedish  OR  is  $1 ,000  per  hour  (Hyltander,  2003) 


Objective  2:  Increased  Access 


•  Good  laparoscopic  skills  cannot  be  developed  by  merely  watching  an  expert. 

•  Laparoscopic  proficiency  is  only  realized  after  sufficient  practice  in  the 
minimally  invasive  environment.”  (Pearson  etai,  2002) 

•  Students  trained  in  VR  are  29%  faster  at  performing  laparoscopic  surgeries 
and  make  up  to  five  times  fewer  mistakes  (Enochssonetai,  2004;  and  Seymour,  2002) 


•  Learning  begins  with  “do  one”  (Jordan  et  al,  2001;  Gallagher  et  al,  2001b;  Madan  &  Frantzides,  2007). 


Objective  3:  Reduced  Time 


•  Lap  simulators  differentiate  experienced  from  inexperienced  users  based 
on  their  performance  scores  with  the  simulator  (Adamsen  et ai,  2005) 


•  MIST-VR  simulator  could  determine  which  students  will  never  achieve 
proficiency  and  should  be  dropped  from  a  training  program  (Gaiiagher  et  ai,  2004) 

•  Non-VR  trained  students  are  nine  times  more  likely  to  fail  to  make 
progress  in  their  performance  than  those  who  use  VR  in  their  training 

(Seymour,  2002) 


Objective  4:  Reduced  Errors 


•  Medical  error  is  responsible  for  between  44,000  and 
98,000  deaths  per  year  <iom,  1999). 

•  Laparoscopic  surgery  has  an  error  rate  that  is  three 
times  higher  than  that  of  open  surgery.  Error  rate  has 
not  been  decreasing  over  an  eight  year  period  as 
surgeons  become  more  experienced  (Huang  etai,  2005). 

•  In  laparoscopy,  observation  does  little  to  convey  the 
skills  that  must  be  mastered.  Only  actual  practice  has 

been  effective  at  this  (Jordan  et  al,  2001 ;  Gallagher  et  al,  2001b;  Madan  & 
Frantzides,  2007). 

•  Simulations  can  improve  the  performance  of  surgeons 
because  they  become  familiar  with  the  appearance  of 
organs  and  tissue  on  a  two  dimensional  computer 

monitor  (Huang  etal,  2005). 
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Misleading  Assumptions  in  Traditional  Education 


•  Assumption  1 :  Didactic  Education  is  Effective 

-  Though  surgeons  or  residents  may  learn  new  information  during  educational 
lectures,  they  do  not  incorporate  it  into  their  practice.  It  has  no  impact  on  their 
actions  in  delivering  medicine.  (Davis  etai  1995  &  1999;  weiier et  ai  2005) 

•  Assumption  2:  Sufficient  Access  to  Faculty  and  Patients  is 
Possible 

-  Availability  of  faculty  is  a  major  limitation  in  medical  education  (Dunkin  et  ai,  2007;  satava, 
2008) 

-  Many  studies  assume  adequate  access  a  priori  (Gerson  &  van  Dam,  2003) 

•  Assumption  3:  Practicing  on  Live  Patients  is  Acceptable 

-  Medical  schools,  faculty,  and  residents  are  finding  new  restrictions  on  the  type  and 
amount  of  training  that  can  be  conducted  with  a  live  patient  (Murphy  et  ai,  2007;  Murray  et  ai, 

2005;  Satava,  2004a;  Ziv  et  ai,  2005). 


Training  Technology  Options 


Human 


Animal 


Box  Trainer 


Mannequin 


VR/Game  Tech 


Human 


Learn  on 
humans: 


Living 

patients,  the 
newly  dead, 
and  cadavers 


Advantage 

Exact  Replica, 
Existing  OR 


Disadvantage 

Scarcity, 
Single  Use, 
Ethical  Issues 


Examples 

Cadavers 
Live  Patients 


Animal 


Learn  on 
animals: 


Living  and 
newly  dead 
pigs,  cats, 
and  others 


Advantage 

Similarities, 

Availability 


Disadvantage 

Anatomy, 
Single  Use, 
Social  Mores 


Examples 

Porcine  Labs 


Box  Trainer 


Learn  on 
organs  in  a 
box: 

Human¬ 
shaped  box 
contains 
organs, 
tissue,  or  test 
devices 


Advantage 

Availability, 
Convenience, 
Human  Shape 


Disadvantage 

Not  Alive, 
Single  Use, 
Animal  Organs 


Examples 

MIC-Trainer 


Mannequin 

Learn  on  a 
physical 
replica: 

A  full-body 
device  with 
synthetic 
skin,  organs, 
and  fluids 


Advantage 

Human  Shape, 
Logistics 


Disadvantage 

Static, 

Lacks  Realism 


Examples 

CPR  Annie 


Simulation 


Learn  on  an 
animated 
machine: 

Includes 
computer, 
hydraulics, 
pneumatics, 
and  electrical 
responses 


Advantage 

Rich  Experience, 
Multi-Function, 
Programmable 


Disadvantage 

High  Cost, 
Complexity 


Examples 

Sim  One 
HPS 


VR/Game 


Learn  on 
computer 
images: 

Mathematical 
models, 
visual 
images, 
sounds,  and 
some  tactile 
feedback 


Advantage 

Rich  Experience, 
Flexibility, 
Low  Cost 


Disadvantage 

Screen-barrier, 

Non-tactile 


Examples 

MJgT-VR 

dV-Trainer 


r 

v 


Complete  Study 


Smith,  R.  (2009).  Game  Technology  in  Medical  Education.  Modelbenders  Press. 
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Robotic  Curriculum 


Curriculum  Development: 

•Define  Robotic  Surgery  outcomes 
•Develop  Robotic  Surgery  curriculum 
•Develop  specific  training  tasks 


Curriculum  Validation: 

•Validate  training  tasks 
•Identify  testing  measures 
•Set  passing  criteria 
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Communication  Latency: 

•Map  surgical  movements  to  latency 
•Redesign  for  latency  tolerance 
•Introduce  instruments  for  safety 
•Target  city-pairs  by  latency 


Automatic  Surgery: 

•Record  movements  in  simulator 
•Execute  movements  with  robot 
•Measure  accuracy  of  outcome 


Surgical  Rehearsal: 

•Patient-specific  rehearsal  simulator 
•Simulated  patient  physiology 
•Measure  impact  on  surgical  perfonn 


Military-use  Validation: 

•Identify  military  constraints 
•Validate  simulator  for  military-use 
•Define  deployable  package 
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Intuitive  Surgical's  Training  Pathway 
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Surgeon  and  OR  Team  Pathway 


Phase 

Content 

Trainer 

1:  Introduction  to  da  Vinci  Surgery 

▼ 

Product  Training 

▼ 

Intuitive  Surgical 

III  Preparation  and  System  Training 

▼ 

III:  Post  System  Training 

T 

IV:  Advanced  Training 
▼ 

Clinical  Training 
▼ 

Independent  Surgeons  & 
Societies/Academic  Institutions 

Beyond  the  Pathway 

Continuing  Clinical  Education 

Independent  Surgeons  &  Societies/ 
Academic  Institutions 

■  Phases  Ml  focus  on  product  training,  while  phases  lll-IV  focus  on  clinical  training 

■  Beyond  the  pathway  skills  are  honed  with  continuing  clinical  education 

•  •  • 

•  «*•••■ 

1 1 1  •  •  •  • 

•  1 1  •  •  •  • 

>#••••• 

•  i  •  i  ■  •  ■ 
i  •  •  •  •  •  • 
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FRS  Mission  Statement 


Create  and  develop  a  validated  multi¬ 
specialty,  technical  skills  competency  based 
curriculum  for  surgeons  to  safely  and 
efficiently  perform  basic  robotic-assisted 
surgery. 


Note:  The  intent  is  to  create  a  curriculum  that  is  device-independent.  This  is 
admittedly  difficult  given  the  single  approved  surgical  robot  at  this  time.  Therefore, 
significant  attention  is  being  paid  to  material  that  is  device-flexible  in  anticipation  of 
future  robots. 
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Participating  Organizations 


American  Association  Gynecologic 
Laparoscopy  (AAGL)+ 

American  College  of  Surgeons  (ACS) 
American  Congress  of  OB-Gyn  (ACOG) 

American  Urologic  Association  (AUA) + 

American  Academy  of  Orthopedic 
Surgeons  (AAOA) 

American  Assn  of  Thoracic  Surgeons 
(AATS) 

American  Assn  of  Colo-Rectal  Surgeons 
(ASCRS) 

American  Assn  of  Gynecologic 
Laparoscopists  (AAGL) 

Florida  Hospital  Nicholson  Center* 

U.S.  Department  of  Defense  (DoD)* 

U.S.  Department  of  Veterans  Health 
Affairs  (VHA) 


Minimally  Invasive  Robotic  Association 
(MIRA)* 

Society  for  Robotic  Surgery  (SRS) 

Society  of  American  Gastrointestinal 
and  Endoscopic  Surgeons  (SAGES) + 

American  Board  of  Surgery  (ABS) 

Accreditation  Council  of  Graduate  Med 
Education  (ACGME) 

Association  of  Surgical  Educators  (ASE) 

Residency  Review  Committee  (RRC)  - 
Surgery 

Royal  College  of  Surgeons-lreland  (RCSI) 
Royal  College  of  Surgeons-London  (RCSL) 

*  Funding  Sources 

+  Executive  Committee 
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Development  of  Curriculum  from  common  template 

“Sweet*  Tree” 


Society  Advanced 

Individual  Procedures 


Society  Core 
Advanced  Skills 


Society  Core 
Basic  Skills 
Template 


Core  Template 

All  societies  agree 


Nephrectomy 


Hysterectomy 


Cystectomy 


FUroRS 


R  Colectomy 


Oophorectomy 


Etc 


Sigmoidectomy 


FGynRS 


FColoRS 


FRS 


Etc 


FUroRS 

FGynRS 

FColoRS 

F???RS..etc 

Advanced 

Advanced 

Advanced 

Advanced 

F???RS..etc 


*  Adapted  from  Rob  Sweet,  MD,  Professor  of  Ur  ology,  Univer  sity  Minnesota,  2010 
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WHO  HOW 


The  Metrics  Drives  the  Process 


urriculum  Development 


< 

z 

3 


Outcomes 

& 

Metrics 


Consensus 

Conference 


ABS 

SAGES 

ACS 

Specialty 

Societies 


Curriculum 

Development 

Simulator 

Development 

Validation 

Studies 

Implement: 

Survey 

Training 

Certification 

Standard 

Curriculum 

Template 

Engineering 

Physical 

Simulator 

Standard 

Validation 

Template 

Current 

Procedures 

SAGES 

ACS 

Societies 

Academia 

Industry 

with 

Academia 
Medical  Input 

ACS 

SAGES, 

Participating 

Societies 

FLS 

SAGES/ACS 

Issue 

Certification 


Issue 

Mandates 

And 

Certificates 


ABS 


certifier 


Creator:  Rick  Satava,  MD,  Univ  of  Washington 
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Consensus  Conference  Process 


1.  Outcomes  Measures  (Dec  12-13,  2011) 

2.  Curriculum  Outline  (April  29-30,  2012) 

2.5  Curriculum  Development  (Aug  17-18,  2012) 

3.  Validation  Criteria  (December,  2012) 

4.  Validation  Studies 

•  •  •  • 

5.  Transition  to  Objective  Testing  Organization 
(est.  July  2013) 


•Expert  Discussion  and  Contributions 
•Modified  Delphi  Voting  Mechanism 
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#1  Outcomes  Measures 


•  •  t  . 

t  •  •  •  • 

|  Pre-Operative 

Intra-Operative 

Post-Operative 

System  Settings 

Energy  Sources 

Transition  to  Bedside  Asst 

Ergonomic  Positioning 

Camera  Control 

Undocking 

Docking 

Clutching 

Robotic  Trocars 

Instrument  Exchange 

OR  Set-up 

Foreign  Body  Management 

Situation  Awareness 

Multi-arm  Control 

Closed  Loop  Comms 

Eye-hand  Instrument  Coord 

Respond  to  System  Errors 

Wrist  Articulation 

Atraumatic  Tissue  Handling 

1 

Dissection  -  Fine  &  Blunt 

Cutting 

Needle  Driving 

Suture  Handling 

Knot  Tying 

9  •  • 

•  •  «  • 

Safety  of  Operative  Field 

57 

Faculty  Members:  Outcomes  Measures 


•  Arnold  Advincula,  MD 

•  Rajesh  Aggarwal,  MD 

•  Mehran  Anvari,  MD 

•  John  Armstrong,  MD 

•  Paul  Neary,  MD 

•  Wallace  Judd,  PhD 

•  Michael  Koch,  MD 

•  Kevin  Kunkler,  MD 

•  Vipul  Patel,  MD 

•  COL  Robert  Rush,  MD 

•  Richard  Satava,  MD 

•  Danny  Scott,  MD 

•  Mika  Sinanan,  MD 

•  Roger  Smith,  PhD 

•  Dimitrios  Stefanidis  MD 

•  Chandru  Sundaram,  MD 

•  Robert  Sweet,  MD 

•  Edward  Verrier,  MD 


American  Assoc  of  Gynecologic  Laparoscopists  &  ACOG 
Royal  College  of  Surgeons  -  London 
Minimally  Invasive  Robotic  Association  (MIRA) 

USF  Health,  CAMLS  (now  Florida  Surgeon  General) 

Royal  College  of  Surgeons  -  Ireland 
Authentic  Testing  Corp. 

American  Board  of  Urology 

US  Army  Medical  Research  &  Materiel  Command  TATRC 
Global  Robotics  Institute  -  Florida  Hospital  Celebration  Health 
US  Army  Madigan  Healthcare  System 
Minimally  Invasive  Robotic  Association  (MIRA) 

Society  of  American  Gastro  and  Endoscopic  Surgeons  (SAGES) 
University  of  Washington 
Florida  Hospital  Nicholson  Center 
Association  for  Surgical  Education 
American  Urological  Association 
American  Urological  Association 
Joint  Council  on  Thoracic  Surgery  Educatiol 
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Skills  Definition  (Sample) 

t 

• 

t 

• 

• 

Task  Name 

Description 

Errors 

Outcomes 

Metrics 

Importance  Rating 

1 

2 

3 

4 

Total 

Score 

Rank 

Order 

• 

Needle 

Accurate  and 

Tearing  tissue, 

Accurate  and 

Time,  accuracy, 

0 

0 

3 

6 

33 

3 

• 

• 

Hrivinp 

efficient 

Troughing  the 

efficient 

tissue  damage, 

• 

manipulation 

needle. 

placement  of 

material  damage 

• 

• 

of  the  needle. 

Needle 

needle  through 

• 

scratching, 

targeted  tissue, 

• 

• 

Wrong  angle 

Following  the 

# 

on  entry/exit. 

curve  of  the 

• 

• 

Adjacent  organ 

needle. 

• 

injury, 

without 

t 

t 

(more) 

associated 

• 

tissue  injury 

• 

• 

Atraumatic 

Haptic 

Traumatic 

Manipulates 

Metric-respect  for 

0 

0 

3 

6 

33 

4 

• 

handling 

comprehensio 

handling, 

tissue  and 

tissue. 

• 

t 

n.  Using 

Tissue  damage 

surgical 

Stress  and  strain 

• 

graspers  to 

or  hemorrhage 

materials 

indentation  and 

t 

• 

hold  tissue  or 

without 

deformation 

• 

surgical 

damage 

t 

• 

material 

• 

without 

• 

• 

crushing  or 

59 

• 

tearing. 

#2  Curriculum  Development 


Didactic  &  Cognitive 

Psychomotor  Skills 

Team  Training 

Lecture-based 

Principle-based 

Checklist-based 

Intro  to  Robotic  System 

Based  on  Physical  Models 
(Virtual  Models  are  Derivative) 

#1:  WHO  Pre-Op 

Pre-Operative  Activity 

3D  Exam  Tools 

#2:  Robotic  Specific 

Intra-Operative  Activity 

Use  Tasks  that  have 

Evidence  of  Validity 

#3:  Undocking  & 
Debriefing 

Post-Operative  Activity 

Multiple  Outcomes 
Measured  per  Exercise 

#4  Crisis  Scenarios 

Each  Activity  includes: 

Goals,  Conditions,  Metrics, 
Errors,  Standards 

Cost  Effective  Solution 

High  Fidelity  for  Testing, 
Lower  Fidelity  for  Training 

IRR  Requires  Ease  of 
Administration 

Florida  hospital 


Nicholson  Center 


Faculty  Members:  Curriculum  Develop 


•  Arnold  Advincula 

•  Abdulla  Al  Ansari 

•  David  Alba  la 

•  Richard  Angelo 

•  James  Borin 

•  David  Bouchier-Hayes 

•  Timothy  Brand 

•  Geoff  Coughlin 

•  Alfred  Cuschieri 

•  Prokar  Dasgupta 

•  Ellen  Deutsch 

•  Gerard  Doherty 

•  Brian  Dunkin 

•  Susan  Dunlow 

•  Gary  Dunnington 

•  Ricardo  Estape 

•  Peter  Fabri 


Vicenzo  Ficarra 
Marvin  Fried 
Gerald  Fried 
Tony  Gallagher 
Piero  Giulianotti 
Larry  Glazerman 
Teodar  Grantcharov 
James  Hebert 
Robert  Holloway 
Santiago  Horgan 
Lenworth  Jacobs 
Arby  Kahn 
Keith  Kim 
Michael  Koch 
Rajesh  Kumar 
Gyunsung  Lee 
Raymond  Leveillee 
Jeff  Levy 


C.Y.  Liu 

Col.  Ernest  Lockrow 
Fred  Loffer 
Guy  Maddern 

Scott  Magnuson 

Javier  Magrina 
Michael  Marohn 
David  Maron 
Martin  Martino 
W.  Scott  Melvin 
Francesco  Montorsi 
Alex  Mottrie 
Paul  Neary 
Eduardo  Parra-Davila 
Vipul  Patel 
Gary  Poehling 
Sonia  Ramamoorthy 
Koon  Ho  Rha 


Richard  Satava 
Steve  Schwaitzberg 
Danny  Scott 
Roger  Smith 
Hooman  Soltanian 
Dimitrios  Stefanidis 
Chandru  Sundaram 
RobertSweet 
Amir  Szold 
Raju  Thomas 
Oscar  Traynor 
Thomas  Whalen 
Gregory  Weinstein 
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Robot 


Simulator 


Testing 


•  • 


Environments 


#3  Validation  Conference 


Criteria 

—  Validate  the  curriculum  and  passing  criteria  that 
will  be  used  to  grant  certification 


Multi-Institutional  Study 

- 10  independent  sites 

-  ACS  AEI  accredited 

-  Faculty  in  at  least  2  specialties 


FLORIDA  HOSPITAL 

Nicholson  Center 


Conclusions 


Objective  curriculum  in  robotic  surgery  is 
:j  needed  for  certification 

Development  of  such  a  curriculum  is 
underway  by  a  multi-specialty  working  group 
of  experienced  surgeons 

iiii* 

Florida  Hospital  is  actively  supporting  this 
effort  with  surgical  experts  and  grant  funding 
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Panel:  Examining  and  Measuring 
Training  Effectiveness 

Roger  Smith,  Florida  Hospital 
Troy  Reihsen,  University  of  Minnesota 
Matthew  Lineberry,  Naval  Air  Warfare  Center 
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Fundamentals  of  Robotic  Surgery 
Curriculum  and  Certification 


Roger  Smith,  PhD 


Chief  Technology  Officer 
roger.smith@flhosp.org 
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www.nicholsoncenter.com 
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Robotic  and 


Robotic  Curriculum 


Consensus  Conference: 

•Define  Robotic  Surgery  outcomes 
•Develop  Robotic  Surgery  curriculum 
•Develop  specific  training  tasks 


Curriculum  Validation: 

•Validate  training  tasks 
•Identify  testing  measures 
•Set  passing  criteria 
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Telesurgery  Research  Project 


Communication  Latency: 

•Map  surgical  movements  to  latency 
•Redesign  for  latency  tolerance 
•Introduce  instruments  for  safety 
•Target  city-pairs  by  latency 


Automatic  Surgery: 

•Record  movements  in  simulator 
•Execute  movements  with  robot 
•Measure  accuracy  of  outcome 


Simulation 


Military-use  Validation: 

•Identify  military  constraints 
•Validate  simulator  for  military-use 
•Define  deployable  package 


Surgical  Rehearsal: 

•Import  patient  CT  scan 
•Patient-specific  rehearsal  simulator 
•Measure  impact  on  surgical  perfonn 
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Grants  Funding  Work 


PI:  Roger  Smith,  PhD  &  Vipul  Patel,  MD 
Florida  Hospital  Nicholson  Center 


PI:  Richard  Satava,  MD 
Minimally  Invasive  Robotics  Assoc 


Source:  Department  of  Defense 


Source:  Intuitive  Surgical  Inc. 


*  This  work  was  supported  by  an  unrestricted  educational  grant  through  the  Minimally  Invasive  Robotics  Association  from  Intuitive 
Surgical  Incorporated. 

**  This  effort  was  also  sponsored  by  the  Department  of  the  Army,  Award  Number  W81XWH-1 1-2-01 58  to  the  recipient  Adventist  Health 
System/Sunbelt,  Inc.,  Florida  Hospital  Nicholson  Center.  “The  U.S.  Army  Medical  Research  Acquisition  Activity,  820  Chandler  Street, 
Fort  Detrick  MD  21702-5014  is  the  awarding  and  administering  acquisition  office.”  The  content  of  the  information  does  not  necessarily 
reflect  the  position  or  the  policy  of  the  Government,  and  no  official  endorsement  should  be  inferred. 
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Mission  Statement 


Create  and  develop  a  validated  multi¬ 
specialty,  technical  skills  competency  based 
curriculum  for  surgeons  to  safely  and 
efficiently  perform  basic  robotic-assisted 
surgery. 
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Consensus  Conference  Process 


Outcomes  Measures  (Dec  12-13,  2011) 
Curriculum  (April  29-30,  2012) 

Validation  Criteria  (est.  July  12-13,  2012) 

•  •  •  • 

Validation  Studies 

•  •  •  »  • 

Transition  to  Objective  Testing  Organization 

■:(est  July  2013) 

•  «  •  • 

•  ••a- 

Expert  Discussion  and  Contributions 

Modified  Delphi  Voting  Mechanism 

•  •  • 

gj|  FLORIDA  HOSPITAL 

•  •  •  ■  HkSbI  Nicholson  Center 


Participating  Organizations 


*  American  Association  Gynecologic 
Laparoscopy  (AAGL)+ 

•  •  •  t 

•  American  College  of  Surgeons  (ACS) 

•  American  Congress  of  OB-Gyn  (ACOG) 

•  •  •  • 

•  American  Urologic  Association  (AUA) + 

•  American  Academy  of  Orthopedic 
Surgeons  (AAOA) 

•  American  Assn  of  Thoracic  Surgeons 
(AATS) 

•  American  Assn  of  Colo-Rectal  Surgeons 
(ASCRS) 

•  American  Assn  of  Gynecologic 
Laparoscopists  (AAGL) 

•\  U.S.  Department  of  Defense  (DoD)* 

•  U.S.  Department  of  Veterans  Health 
Affairs  (VHA) 


Minimally  Invasive  Robotic  Association 
(MIRA)* 

Society  for  Robotic  Surgery  (SRS) 

Society  of  American  Gastrointestinal 
and  Endoscopic  Surgeons  (SAGES) + 

American  Board  of  Surgery  (ABS) 

Accreditation  Council  of  Graduate  Med 
Education  (ACGME) 

Association  of  Surgical  Educators  (ASE) 

Residency  Review  Committee  (RRC)  - 
Surgery 

Royal  College  of  Surgeons-lreland  (RCSI) 
Royal  College  of  Surgeons-London  (RCSL) 

*  Funding  Sources 

+  Executive  Committee 
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Faculty  Members:  Outcomes  Measures 


Arnold  Advincula,  MD 
Rajesh  Aggarwal,  MD 
Mehran  Anvari,  MD 
John  Armstrong,  MD 
Paul  Neary,  MD 
Wallace  Judd,  PhD 
Michael  Koch,  MD 
Kevin  Kunkler,  MD 
Vipul  Patel,  MD 
COL  Robert  Rush,  MD 
Richard  Satava,  MD 
Danny  Scott,  MD 
Mika  Sinanan,  MD 
Roger  Smith,  PhD 
Dimitrios  Stefanidis  MD 


AAGL&  ACOG 

Royal  College  of  Surgeons 

Minimally  Invasive  Robotic  Association  (MIRA) 

USF  Health,  CAMLS 

Royal  College  of  Surgeons  -  Ireland 

Authentic  Testing  Corp. 

American  Board  of  Urology 

US  Army  Medical  Research  &  Materiel  Command  TATRC 
Global  Robotics  Institute  -  Florida  Hospital  Celebration  Health 
US  Army  Madigan  Healthcare  System 
Minimally  Invasive  Robotic  Association  (MIRA) 

Society  of  American  Gastro  and  Endoscopic  Surgeons  (SAGES) 


University  of  Washington 
Florida  Hospital  Nicholson  Center 
Association  for  Surgical  Education 
Chandru  Sundaram,  MD  American  Urological  Association 
Robert  Sweet,  MD  American  Urological  Association 

Edward  Verrier,  MD 


Joint  Council  on  Thoracic  Surgery  Educatio 
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#1  Outcomes  Measures 


•  •  t  . 

t  •  •  •  • 

|  Pre-Operative 

Intra-Operative 

Post-Operative 

System  Settings 

Energy  Sources 

Transition  to  Bedside  Asst 

Ergonomic  Positioning 

Camera  Control 

Undocking 

Docking 

Clutching 

Robotic  Trocars 

Instrument  Exchange 

OR  Set-up 

Foreign  Body  Management 

Situation  Awareness 

Multi-arm  Control 

Closed  Loop  Comms 

Eye-hand  Instrument  Coord 

Respond  to  System  Errors 

Wrist  Articulation 

Atraumatic  Tissue  Handling 

1 

Dissection  -  Fine  &  Blunt 

Cutting 

Needle  Driving 

Suture  Handling 

Knot  Tying 

9  •  • 

•  •  «  • 

Safety  of  Operative  Field 
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Faculty  Members:  Curriculum  Develop 


Arnold  Advincula 
Abdulla  Al  Ansari 

David  Albala 

•  •  • 

Richard  Angelo 
James  Borin 

i  «  ■  • 

David  Bouchier-Hayes 

•  •  • 

Timothy  Brand 

•  •  • 

Geoff  Coughlin 

•  •  » 

Alfred  Cuschieri 

Prokar  Dasgupta 

•  •  •  * 

Ellen  Deutsch 

•  •  ■ 

Gerard  Doherty 

Brian  Dunkin 

•  •  • 

Susan  Dunlow 

Gary  Dunnington 

•  •  •  • 

Ricardo  Estape 

•  •  •  • 

Peter  Fabri 


Vicenzo  Ficarra 
Marvin  Fried 
Gerald  Fried 
Tony  Gallagher 
Piero  Giulianotti 
Larry  Glazerman 
Teodar  Grantcharov 
James  Hebert 
Robert  Holloway 
Santiago  Horgan 
Lenworth  Jacobs 
Arby  Kahn 
Keith  Kim 
Michael  Koch 
Rajesh  Kumar 
Gyunsung  Lee 
Raymond  Leveillee 
Jeff  Levy 


C.Y.  Liu 

Col.  Ernest  Lockrow 
Fred  Loffer 
Guy  Maddern 
Scott  Magnuson 
Javier  Magrina 
Michael  Marohn 
David  Maron 
Martin  Martino 
W.  Scott  Melvin 
Francesco  Montorsi 
Alex  Mottrie 
Paul  Neary 
Eduardo  Parra-Davila 
Vipul  Patel 
Gary  Poehling 
Sonia  Ramamoorthy 
Koon  Ho  Rha 


Richard  Satava 
Steve  Schwaitzberg 
Danny  Scott 
Roger  Smith 
Hooman  Soltanian 
Dimitrios  Stefanidis 
Chandru  Sundaram 
RobertSweet 
Amir  Szold 
Raju  Thomas 
Oscar  Traynor 
Thomas  Whalen 
Gregory  Weinstein 
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#2  Curriculum  Development 


Didactic  &  Cognitive 

Psychomotor  Skills 

Team  Training 

Lecture-based 

Principle-based 

Checklist-based 

Intro  to  Robotic  System 

3D  Exam  Tools 

#1:  WHO 

Pre-Operative  Activity 

Multiple  Outcomes  in  Each 
Exercise 

#2:  Robotic  Specific 

Intra-Operative  Activity 

Cost  Effective 

#3:  Undocking  & 
Debriefing 

Post-Operative  Activity 

High  Fidelity  for  Testing, 
Lower  Fidelity  for  Training 

#4  Crisis  Scenarios 

Each  Activity  includes: 

Goals,  Conditions,  Metric, 
Errors,  Standard 

IRR  Requires  Ease  of 
Administration 

Based  on  Specification  of 
Generic  Physical  Models 
(Virtual  Models  are 
Derivative) 

Use  Tasks  that  have 

Evidence  of  Validity 

Nicholson  Center 


Testing  Environments 
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Validation  Conference 


Criteria 


—  Validate  the  exercises  and  exams  that  will  be  used 
to  grant  certification 


Multi-Institutional  Study 


- 10  independent  sites 

—  ACS  AEI  accredited 

-  Faculty  in  at  least  2  specialties 


Florid/7  hospital 

Nicholson  Center 


Download  Reports,  Papers,  and  Presentations 


http://www.simulationfirst.com/frs 
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Thank  You! 


FLORIDA  HOSPITAL 

Nicholson  C  enter 


Interservice/Industry  Training,  Simulation,  and  Education  Conference  ( I/1TSEC )  2012 


Using  Simulators  to  Measure  Communication  Latency  Effects 

in  Robotic  Telesurgery 


Roger  Smith,  PhD 
Florida  Hospital  Nicholson  Center 
Celebration,  FL 
roger  .smith  @  flhosp.org 


Sanket  Chauhan,  MD 
University  of  Minnesota  Medical  School 
Minneapolis,  MN 
sanket@umn.edu 


ABSTRACT 

Robotic  surgical  technology  was  originally  developed  by  the  US  Army  and  DARPA  as  a  tool  to  enable  telesurgery 
at  a  distance.  The  Intuitive  da  Vinci  system  now  provides  a  robotic  surgical  tool  in  a  traditional  operating  room.  But 
research  continues  into  the  extension  of  this  capability  to  patients  that  are  remote  from  the  surgeon’s  location.  In  this 
paper  we  describe  the  interim  results  of  experiments  into  the  effects  of  communication  latency  in  the  safe  execution 
of  robotic  telesurgeries.  These  experiments  were  carried  out  with  the  Mimic  dV-Trainer,  a  simulator  of  the  da  Vinci 
robot,  which  was  configured  to  insert  defined  levels  of  latency  into  the  visual  and  command  data  streams  between  a 
surgeon  and  the  operating  field.  Subjects  were  asked  to  perform  four  basic  robotic  surgical  exercises.  They  were 
allowed  to  rehearse  these  in  a  zero  latency  environment  and  with  a  randomly  assigned  latency  between  100ms  and 
1,000ms.  Then  each  subject  performed  each  exercise  for  measurement  and  analysis  in  our  research. 

This  experiment  measured  the  degradation  of  human  surgical  performance  across  a  range  of  latency  conditions.  This 
paper  reports  on  the  comparison  of  the  level  of  experience  of  the  surgeons  with  their  performance  in  a  latency- 
effected  environment.  The  data  collected  thus  far  refutes  our  hypothesis  that  more  experienced  surgeons  would  be 
more  successful  at  managing  the  effects  of  latency  and  would  perform  better  than  those  with  less  experience. 
Subjects  in  our  experiment  show  no  correlation  between  experience  and  successful  performance  under  latency.  The 
ability  to  manage  latency  in  tele-operations  may  be  shared  between  remote  surgery  and  the  control  of  a  remotely 
piloted  UAV's  and  UGV's.  The  results  of  our  experiments  may  suggest  that  experience  as  a  traditional  pilot  does  not 
necessarily  contribute  to  useful  skills  in  flying  UAV's  or  driving  UGV's  when  latency  is  present. 
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BACKGROUND 

Robotic  surgery  has  been  the  topic  of  science  fiction 
and  scientific  research  for  decades.  As  early  as  1942, 
Robert  A.  Heinlein  published  the  story  “Waldo”  in 
Astounding  Science  Fiction.  He  described  the  use  of 
gloves  and  a  harness  to  allow  Waldo  Jones  to  control 
mechanical  arms  of  any  size  from  large  industrial  and 
construction  equipment  to  miniature  tools  for 
electronic  and  surgical  work.  The  Industrial  Revolution 
gave  us  many  of  the  tools  needed  to  extend  the 
capabilities  of  the  human  body,  but  the  Information 
Age  gave  us  the  computerized  control  systems 
necessary  to  effectively  manipulate  these  devices. 
Surgical  robots  are  a  marriage  of  mechanical, 
electrical,  optical,  and  software  systems  that  can 
empower  a  human  surgeon  to  peer  into  a  patient’s  body 
with  magnified  stereo  vision,  probe  the  internal  organs, 
and  perform  effective  surgery  without  fully  opening  the 
patient’s  body. 

In  1985,  the  PUMA  560  was  used  to  accurately  place  a 
needle  for  a  brain  biopsy  using  CT  guidance  (Kwoh  et 
al,  1988).  In  1988,  the  PROBOT  at  Imperial  College 
London,  was  used  to  perform  prostate  surgery.  In  1992, 
Integrated  Surgical  Systems  introduced  ROBODOC  to 
mill  precise  fittings  in  the  femur  for  hip  replacement. 
Intuitive  Surgical  leveraged  the  research  work  of  the 
Defense  Advanced  Research  Projects  Agency 
(DARPA)  and  used  those  technologies  to  create  the  da 
Vinci  Surgical  System  which  they  introduced  in  1997. 
Computer  Motion  followed  a  similar  path  and  fielded 
the  AESOP  and  ZEUS  robotic  systems  (Figure  1), 
which  were  later  acquired  by  Intuitive  Surgical 
(Satava,  1998;  FDA,  2005). 
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Minneapolis,  MN 
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Figure  1.  ZEUS  Surgical  Research  Robot 


Intuitive  Surgical’s  da  Vinci  robot  is  currently  the  only 
FDA  approved  device  for  robotic  surgery  on  human 
patients.  This  system  senses  the  surgeon’s  hand 
movements  and  translates  them  into  scaled-down 
micro-movements  to  manipulate  tiny  instruments 
inside  the  body.  It  also  detects  and  filters  out  any 
tremors  in  the  hand  movements,  so  that  they  are  not 
expressed  robotically.  The  camera  used  in  the  system 
provides  a  true  stereoscopic  picture  transmitted  to  and 
viewed  through  a  surgeon's  console  (Figure  2). 

These  devices  opened  the  door  for  the  realization  of 
surgery-at-a-distance,  a.k.a.  telesurgery,  in  which  a 
surgeon  is  able  to  extend  his  reach  and  perform 
surgical  procedures  at  a  significant  distance  from  the 
patient.  This  capability  has  been  demonstrated  under 
unique  conditions  by  multiple  experiments  (Himpens, 
1998;  Janetschek,  1998;  Fabrizio,  2000;  Sterbis,  2007). 
Our  research  project  at  the  Florida  Hospital  Nicholson 
Center  is  demonstrating  the  maturity  of  the  existing 
telecommunication  infrastructure  within  a  hospital 
system  to  support  daily,  on-demand  telesurgery  right 
now.  Our  experiments  are  based  on  the  da  Vinci 
surgical  robot  (Intuitive  Surgical,  Inc.)  and  the  dV- 
Trainer  simulator  (Mimic  Technologies,  Inc.). 
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Figure  2.  da  Vinci  Surgical  Robot  (Intuitive  Surgical,  Inc.) 


METHODS 

We  explore  the  effects  of  communication  latency  on 
surgeon  performance.  This  latency  effect  is  created 
using  the  dV-Trainer  simulator  (Figure  3)  of  the  da 
Vinci  surgical  robot  (Hung,  2011;  Kennedy  2009).  The 
simulator  allows  the  insertion  of  specific  levels  of 
controlled  latency  so  that  the  user’s  physical 
movements  are  not  manifest  by  the  simulated 
instruments  until  after  the  defined  latency  period  has 
elapsed. 


Figure  3.  dV-Trainer  Simulator  (Mimic 
Technologies,  Inc.) 


During  actual  telesurgery,  the  messages  sent  between 
the  surgeon's  machine  and  the  remote  patient  station 
will  be  delayed  due  to  the  speed  of  light  and  the 
message  routing  that  occurs  on  the  internet. 
Determining  how  much  latency  can  be  safely  tolerated 
in  surgery  is  an  important  question  (Anvari,  2005  and 
2007).  This  experiment  hypothesizes  that  there  are  two 


distinct  thresholds  of  performance  under  increasing 
latency.  The  first  is  the  level  of  latency  at  which  a 
surgeon  can  first  detect  that  his  or  her  movements  are 
being  affected  by  the  communication  link.  Any 
communication  latency  lower  than  this  level  is 
imperceptible  and  potentially  non-invasive  to  the 
surgical  procedure.  Hence,  if  such  levels  can  be 
achieved  in  the  real  world,  then  telesurgery  may  be 
safe  for  human  surgery  right  now.  The  second  level  is 
the  point  at  which  the  surgeon's  performance  is 
degraded  to  the  point  that  the  surgery  cannot  be 
performed  safely  (Marescaux,  2002;  Lum,  2009).  This 
level  is  identified  through  both  simulator  measured 
performance  and  the  expert  opinion  of  the  surgeon. 
Between  the  first  and  second  thresholds,  a  surgeon  may 
be  able  to  successfully  control  the  effects  of  latency 
and  perform  a  safe  and  successful  procedure.  Beyond 
the  second  threshold,  telesurgery  would  be  considered 
unsafe  with  the  available  equipment  (Figure  4). 


Figure  4.  Conceptual  Diagram  of  Communication 
Latency  Thresholds. 
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Figure  5.  Simulated  Surgical  Skills  Tasks 


We  further  hypothesize  that  more  experienced 
surgeons  will  be  more  successful  at  managing  the 
effects  of  latency  and  would  be  the  best  practitioners 
for  this  extension  of  robotic  surgery.  If  this  hypothesis 
is  correct,  then  surgeons  with  more  experience  should 
achieve  higher  scores  and  shorter  completion  times  in 
the  simulation  experiment  that  we  are  performing.  This 
paper  reports  on  the  analysis  of  this  specific  question 
comparing  surgeon  experience  to  the  ability  to 
successfully  manage  the  effects  of  latency. 

In  this  experiment,  subjects  performed  the  four 
simulated  surgical  skills  exercises  shown  in  Figure  5. 
These  represent  many  of  the  core  skills  that  are 
required  in  robotic  surgery.  Each  subject  performed 
each  exercise  three  times.  First,  the  subject  was  given 
an  opportunity  to  perform  the  task  without  any 
imposed  latency.  This  baseline  insured  that  they  were 
able  to  successftdly  operate  the  controls  under  normal 
conditions.  Second,  they  were  allowed  to  perform  each 
of  the  four  exercises  at  their  randomly  assigned  latency 
level.  These  repetitions  provided  the  learning  necessary 
to  achieve  a  sustained  level  of  proficiency  within  a 
latent  environment  (Rayman  et  al  2006).  Finally,  each 
subject  performed  all  four  exercises  at  the  same 


randomly  assigned  latency  level  and  their  performance 
was  measured  for  analysis  in  the  study. 

A  single,  constant  latency  level  between  100 
milliseconds  (ms)  and  1,000ms  at  increments  of  100ms 
was  randomly  assigned  to  each  subject  (e.g.  100ms, 
200ms,  300ms.  400ms,  etc.).  A  proctor  was  available  to 
instruct  subjects  in  the  use  of  the  equipment  and  to 
guide  them  through  the  curriculum  of  the  protocol. 
However,  this  proctor  was  not  allowed  to  give 
suggestions  on  performance  of  the  exercises  or  to  tell 
the  subject  the  specific  level  of  latency  that  they  were 
experiencing. 

Data  Collection 

Experimental  data  was  collected  by  the  simulator 
software  and  manually  via  questionnaires.  Research 
proctors  administered  a  Pre-Test  questionnaire  on  the 
level  of  surgical  experience  and  related  activities  of  the 
subject.  All  personal  and  performance  data  was 
anonymized  to  insure  that  the  identity  of  the  subject 
could  not  be  linked  to  the  data  that  was  collected.  The 
proctors  also  administered  a  Post -Test  questionnaire  at 
the  conclusion  of  each  of  the  skills  exercises  during  the 
final  performance  stage.  The  simulator  software 
automatically  collected  multiple  measures  of  the 
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subject’s  performance.  This  provided  data  for  all 
subjects  at  zero  latency,  during  their  familiarization 
stage  with  latency,  and  during  the  final  stage  which  is 
the  focus  of  the  analysis.  This  data  will  allow  us  to 
perform  multiple  analyses  of  the  skills  of  robotic 
surgeons  both  with  and  without  communication 
latency,  which  will  be  published  in  future  papers. 

Pre-Test  Questionnaire 

The  Pre-Test  questionnaire  identified  multiple  items  of 
demographic,  experience,  and  practice  data  on  the 
subjects.  These  included:  age,  gender,  dominant  hand, 
surgical  status,  years  of  surgical  experience,  years  of 
laparoscopic  experience,  years  of  robotic  experience, 
number  of  weekly  procedures  in  laparoscopy  and 
robotics,  and  experience  with  laparoscopic  and  robotic 
simulators,  as  well  as  with  video  games  and  musical 
instruments.  Additional  questions  captured  their 
opinion  on  the  use  of  simulation  in  surgical  education 
and  certification. 

This  data  was  then  matched  to  the  data  from  their 
performance  in  the  simulator. 

Simulator  Performance 

During  the  experiment,  the  simulator  itself  collected  a 
number  of  data  points  on  each  subject’s  performance. 
These  included:  time  to  complete,  overall  score,  total 
hand  motion  in  centimeters,  master  working  space, 
number  of  instrument  collisions,  number  of  items 
dropped,  excessive  instrument  force,  distance 
instruments  out  of  view,  incorrect  use  of  electrical 
energy,  simulated  blood  loss,  and  number  of  broken 
blood  vessels. 

Post-Test  Questionnaire 

As  the  subjects  completed  their  final  repetition  of  each 
of  the  four  skills  exercises,  the  proctor  administered  a 
post-test  questionnaire  which  asked  the  subject  for  their 
opinion  on  the  stress  induced  by  the  simulation  with 
latency.  This  included  measures  of  the  mental  and 
physical  demands  of  the  task,  the  pace  of  the  task,  their 
opinion  on  their  level  of  success,  the  amount  of  effort 
expended,  the  level  of  mental  discouragement 
experienced,  and  their  perceived  complexity  of  the 
exercise. 

RESULTS 

This  paper  reports  on  the  analysis  of  data  from  the  first 
54  subjects  in  the  study.  Of  the  54  subjects  who  began 
the  experiment,  several  were  unable  to  complete  all  of 
the  tasks  due  to  the  limited  amount  of  time  that  they 
could  devote  to  the  experiment.  Others  found  the 


experiment  too  taxing  and  elected  to  terminate  their 
participation  before  completion.  As  a  result,  we 
collected  complete  data  sets  without  latency  on  42 
subjects  and  complete  data  with  latency  on  only  21  of 
those  subjects. 

This  data  was  analyzed  to  determine  the  level  of 
correlation  between  the  subjects’  experience  and  their 
performance  both  with  and  without  latency.  For  the 
non-latency  sample  size  of  42  and  a=0.05,  the  Pearson 
Product  Moment  Correlation  (PPMC)  value  is  0.304. 
This  means  that  for  a  correlation  coefficient  of  two 
variables  in  this  size  of  sample  to  be  significant,  it  must 
be  larger  than  the  PPMC  value. 


Table  1.  Correlation  Coefficients  without  Latency 


Exercise 

Overall 

Score 

Time 

Complete 

Pegboard  1 

0.141 

-0.110 

Camera  T argeting 

0.201 

-0.173 

Thread  the  Rings 

0.156 

-0.225 

Energy  Dissection 

0.267 

-0.217 

In  an  environment  without  any  latency  imposed  we 
found  a  positive  correlation  between  years  of  robotic 
experience  and  overall  performance  score,  as  well  as  a 
negative  correlation  between  experience  and  the  total 
time  to  complete  the  exercise  (Table  1).  Both  of  these 
indicate  that  more  experience  leads  to  better 
performance  in  the  simulator.  Though  this  correlation 
is  consistently  supportive  that  surgeons  with  more 
experience  perform  non-latency  exercises  better  than 
those  with  less  experience,  the  degree  of  this 
correlation  is  not  large  enough  to  be  statistically 
significant  for  this  sample  size. 

When  latency  is  added,  a  simple  correlation  coefficient 
is  not  sufficient  for  analyzing  the  effect  of  robotic 
experience  on  performance.  Each  subject  received  a 
randomly  assigned  latency,  of  which  there  were  10 
possibilities.  Within  the  current  sample,  we  have 
between  0  and  5  subject  data  points  at  each  latency 
level.  Therefore,  under  latency,  we  examine  the  data  by 
visual  examination  of  a  multiline  scatter  plot. 

Scatterplots  can  illustrate  the  linear  relationship 
between  two  variables  in  the  model.  Without  latency,  a 
relationship  can  be  seen  for  both  overall  performance 
score  and  time  to  complete  the  exercise  (Figures  6  & 
7).  Flowever,  when  latency  is  present,  the  plots  show 
that  there  is  not  a  relationship  between  the  two 
variables  for  the  subjects  tested  (Figures  8  &  9). 
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Pegboard  without  Latency 


Robotic  Experience  (Years) 


Figure  6.  Correlation  between  Robotic  Experience 
and  Overall  Score  for  the  Peg  Board  exercise 
without  communication  latency. 


Figure  8.  Correlation  between  Robotic  Experience 
and  Overall  Score  for  the  Peg  Board  exercise  with 
various  communication  latencies. 


The  data  suggests  that  surgeons  who  have  more 
experience  in  robotic  surgery  are  not  better  equipped  to 
self-manage  the  challenges  presented  by 
communication  latency  in  telesurgery.  Subjects  with 
little  experience  are  as  likely  to  successfully  manage 
latency  as  are  surgeons  with  more  experience. 

This  same  trend  holds  when  comparing  independent 
variables  like  total  surgical  experience  and 
laparoscopic  experience  to  the  scores  achieved  in  the 
simulator  with  latency. 

CONCLUSIONS 

The  lack  of  correlation  between  experience  and 
telesurgical  performance  under  latency  refutes  our 
original  hypothesis  that  a  more  experienced  surgeon 
would  more  successfully  manage  the  effects  of  latency. 
This  negative  finding  has  led  to  speculation  on  the 
cause  of  these  results.  Several  may  be  possible,  but 
each  will  require  additional  experimentation.  First, 
experienced  surgeons  may  be  very  talented,  but  fixed, 
in  their  methods  of  performing  surgery.  This  may  lead 
them  to  perform  poorly  under  latency  because  it  is 
difficult  for  them  to  modify  their  behaviors,  where 
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Figure  7.  Correlation  between  Robotic  Experience 
and  Time  to  Complete  for  the  Peg  Board  exercise 
without  communication  latency. 
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Figure  9.  Correlation  between  Robotic  Experience 
and  Time  to  Complete  for  the  Peg  Board  exercise 
with  various  communication  latencies. 


inexperienced  surgeons  are  less  ingrained  and  more 
adaptable  to  the  situation.  Second,  since  the  simulator 
is  a  computer-generated  virtual  environment,  it  is 
possible  that  surgeons  who  have  more  experience  in 
simulators,  virtual  worlds,  and  computer  games  may 
have  developed  a  proficiency  for  solving  problems  in 
this  kind  of  environment.  They  may  also  have 
experienced  latency  in  those  environments  and 
developed  techniques  for  compensating  for  it.  Third, 
the  ability  to  manage  latency  may  be  related  to  the 
physical  and  biological  wiring  of  an  individual.  This 
could  be  a  similar  phenomenon  to  the  tendency  for 
some  people  to  experience  simulator  sickness,  while 
others  do  not  suffer  from  it.  These  speculations  are 
worthy  of  further  investigation. 

The  objective  of  this  analysis  was  to  identify  the  degree 
to  which  a  surgeon  can  compensate  for  the  effects  of 
latency  that  are  present  in  a  telesurgery  environment. 
The  long-term  goal  is  to  identify  the  thresholds  where 
safe  and  successful  surgery  can  be  performed.  Our 
findings  at  this  point  refute  our  hypothesis  that  more 
experienced  surgeons  would  be  able  to  manage  latency 
more  successfully.  In  the  data  collected  there  is  no 
correlation  between  robotic  experience  and  the  ability 
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to  achieve  a  higher  score  in  the  simulator  when  latency 
is  inserted  into  the  procedure. 

These  results  may  inform  research  on  remote 
teleoperation  in  other  environments,  such  as  the  control 
of  UAV’s  and  UGV’s.  Experienced  pilots  and  vehicle 
drivers  may  not  be  better  equipped  to  manage  the 
effects  of  latency  than  pilots/drivers  with  less 
experience.  Other  factors  may  be  more  important  in 
predicting  a  person's  ability  to  tele-operate  a  remote 
system  successfully.  The  similarity  between  remote 
surgery  and  remote  vehicle  operation  is  speculative  and 
would  require  specific  research  experiments  with  those 
systems  to  verify. 
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Robotic  and  Telesurgery  Research  using  Simulation 


Telesurgery 


Comms  Latency: 

•Modify  surgical  procedures 
•Safe  Telesurgery  at  500ms 
•Match  to  City-Pairs 


Automatic  Surgery: 

•Record  Surgery  in  Simulator 
•Execute  with  Unmanned  Robot 
•Identify  Control  Variables 


Simulation 


Surgical  Rehearsal: 

•Dynamic  Organ  Model  in  Sim 
•Patient-specific  Rehearsal 
•Improve  Surgeon  Performance 


Military-use  Validation: 

•Simulator  of  Robotic  Surgery 
•Retain  Skills  in  Theater 
•Define  Deployable  Package 


Robotic  Curriculum 


Consensus  Conferences: 

•Define  Certification  Criteria 
•Develop  Curriculum 
•Develop  Training  Tasks 


Curriculum  Validation: 

•Validate  the  Program 
•Identify  Testing  Measures 
•Set  Passing  Criteria 


w 


Telesurgery:  Communication  Latency 


•Stereo  HD  Video 
•Team  Audio 


•Robot  Commands 
•Surgeon  Audio 


* 


Comm  Latency  = 


Telesurgery:  Simulate 


da  Vinci  Skills  Simulator 


Mimic  dV-Trainer 


Simulation:  Surgical  Rehearsal 


Telesurgery:  Automatic  Surgery 


Pierce  the  object  at  the  indicated  location  and  pull  the 
redle  through.  Start  from  the  yellow  aide  of  the  target 


Industry  Perspective 


*  Simulation  as  a  Research  Lab 


-  Simulated  environments  are  a  viable  and  affordable  research 
environment  within  which  to  conduct  experiments. 

*  Simulation  for  Rehearsal 


-  Simulation  is  a  tool  for  real-time  preparation  for  surgery. 

*  Simulation  for  Education 


-  Redesigning  GME  surgical  courses  to  include  simulators  along  with 
classroom  and  laboratory  components. 
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Serious  Games 
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Technical,  Social  and  Economic  importance  of  simulation  and  gaming 
Focus  on  techniques,  tools  and  technologies 
Historical  summary  and  future  possibilities 

Explores  and  contrasts  Military  and  Commercial  gaming  evolutions 


Roger  D.  Smith 

Author  of  Cmm •  Todnotff  for  MmI Uml  UiKfhn 
•nd  StaiaiillM  ImterwfftrmMlty 


•  The  traditional  Halstedian  apprenticeship  model  of  'see  one,  do  one,  teach  one1  is  no 
longer  adequate  to  train  surgeons,  since  good  laparoscopic  skills  cannot  be 
developed  by  merely  watching  an  expert."  -  A.  Pearson,  M.D. 

•  "There  is  no  excuse  for  the  surgeon  to  learn  on  the  patient."  -  William  Mayo,  1927 

•  Dr.  Smith’s  Book  proposes  4  hypotheses: 

1 .  Virtual  Reality  and  gaming  can  reduce  costs  for  surgical  training 

2.  VR  and  gaming  can  improve  repetitive  practice  to  assess  patient  symptoms 

3.  VR  and  game  training  environments  can  reduce  training  times  (for  equal  skill) 

4.  VR  and  gaming  can  reduce  medical  errors 
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Game  Technology  in 
Medical  Education: 

An  Inoglry  into  Ihc  Effectiveness 
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Dr.  Roger  Smith 


Florida  Hospital 


•8  Campus  Hospital  System  in  Orlando,  Florida 
•34  Regional  Campuses  across  Florida 
•2,188  beds 

•Largest  provider  of  healthcare  in  Florida 

•Largest  by  some  measures  in  the  entire  United  States 


Nicholson  Center 


*  Surgical  Education 

-  Robotic  Surgery 

-  Laparoscopic  Techniques 

-  Orthopedic  Equipment 


*  Surgical  Research 

-  Robotic  &  Telesurgery 

-  Surgical  Education 

-  Automatic  Surgery 
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Errors  Eliminate  Profits 


*  Minor  Complication 

-  Revisit  eliminates  all  profit  from  the  original  surgery 

*  Major  Complication 

-  Revisit  costs  3X  the  profit  from  the  original  surgery 
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\  Creating  Experts  &  Eliminating  Errors 


Outliers 

e 

The  Story  of  Success 

Malcolm 

Gladwell 


10,000  hours  to  become  an  expert  -  Gladwell 
“There  is  no  excuse  for  the  surgeon  to  learn  on  the  patient.”  -  William  Mayo,  1927 
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Medical  Education  -  Explosion  of  Information 


•  Medical  procedures  are  becoming  more  numerous  and  more 
complex  -  medical  knowledge  has  “hypertrophied”  (cooked 


•  Training  residents  to  a  common  level  of  knowledge  and 

competence  is  already  impossible  (Satava,  2008) 
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“The  Perfect  Storm” 


•  Risk  to  patient  health.  (McDougall, 

2007) 

•  Ethics  of  practicing  on 

patients.  (Satava,  2004;  Murphy,  2007) 

•  Cost  is  a  barrier  to  training. 

(Bridges,  1999) 

•  Insurance  coverage  of 
educational  actions.  <satava,  2004) 

•  Working  hour  limits.  (Satava,  2004) 

•  Availability  of  training 

opportunities.  (Birden,  2007;  Davis,  1999) 

•  Access  to  training.  (Dunkin,  2007;  Spitzer, 

1997) 


,  2007) 


•  Complexity  of  modern  surgery. 

(McDougall,  2007) 

•  Volume  of  unique  procedures. 

(Reznick  &  MacRae,  2006) 

•  Proficiency-based  Medicine. 

(Murray,  2005) 

•  Quality  of  technology.  (Murphy,  2007) 

•  Expectations  around  computer 

technologies.  (Murphy,  2007) 

•  Acceptance  of  technology.  <ziv, 

2003) 

•  Learning  from  Mistakes.  (Ziv,  2005) 


Intuitive  Surgical’s  da  Vinci  Robot 


Video  at:  ittp://www.voutube.com/watch?v=ONZLpWrJGqk 
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5otic  and  Telesurgery  Research  using  Simulation 


Comms  Latency: 

•Modify  surgical  procedures 
•Safe  Telesurgery  at  500ms 
•Match  to  City-Pairs 


Automatic  Surgery: 

•Record  Surgery  in  Simulator 
•Execute  with  Unmanned  Robot 
•Identify  Control  Variables 

Simulation 


Surgical  Rehearsal: 

•Dynamic  Organ  Model  in  Sim 
•Patient-specific  Rehearsal 
•Improve  Surgeon  Performance 


Military-use  Validation: 

•Simulator  of  Robotic  Surgery 
•Retain  Skills  in  Theater 
•Define  Deployable  Package 


Robotic  Curriculum 


Consensus  Conferences: 


•Define  Certification  Criteria 
•Develop  Curriculum 
•Develop  Training  Tasks 


Curriculum  Validation: 

•Validate  the  Program 
•Identify  Testing  Measures 
•Set  Passing  Criteria 
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Telesurgery:  Simulated  Latency 
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da  Vinci  Skills  Simulator 


Mimic  dV-Trainer 


Simulation:  Surgical  Rehearsal 


Telesurgery:  Automatic  Surgery 
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*  Simulation  as  a  Research  Lab 

-  Simulated  environments  are  a  viable  and  affordable  research 
environment  within  which  to  conduct  experiments. 

*  Simulation  for  Rehearsal 

-  Simulation  is  a  tool  for  real-time  preparation  for  surgery. 

*  Simulation  for  Education 

-  Redesign  GME  surgical  courses  to  include  simulators  along  with 
classroom  and  laboratory  components. 
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Abstract.  To  standardize  the  curriculum  and  certification  of  robotic  surgeons,  a  series  of  consensus 
conferences  have  been  used  to  compile  the  outcomes  measures  and  curriculum  that  should  form  the 
basis  for  the  Fundamentals  of  Robotic  Surgery  (FRS)  program.  This  has  resulted  in  the  definition  of 
25  specific  outcomes  measures  and  the  creation  of  curriculum  for  teaching  those  via  didactic 
lecture,  psychomotor  skills  labs,  and  team  training  activities.  This  work  has  been  supported  and/or 
reviewed  by  the  leading  surgical  societies  involved  in  the  use  of  robotic  surgery. 


Introduction 

In  2004,  the  Society  of  American  Gastrointestinal  and 
Endoscopic  Surgeons  (SAGES)  launched  the  validated 
Fundamentals  of  Laparoscopic  Surgery  (FLS)  curriculum 
and,  together  with  the  American  College  of  Surgeons 
(ACS),  promoted  the  FLS  as  a  minimum  standard  before 
a  surgeon  should  be  allowed  to  perform  laparoscopic 
procedures  independently  [1],  In  2009,  The  American 
Board  of  Surgery  (ABS)  mandated  that  in  addition  to 
Advanced  Cardiac  Life  Support  (ACLS)  and  Advanced 
Trauma  Life  Support  (ATLS)  a  certificate  documenting 
the  successful  passing  of  the  FLS  exam  be  included  in  the 
application  in  order  to  be  eligible  to  sit  the  examination 
for  certification  in  General  Surgery  [2]. 
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Figure  1.  Growing  number  of  robotic  surgical 
procedures 

Source:  Intuitive  Surgical,  Inc  Investor  Prospectus,  Feb,  2012 


During  the  last  decade,  robotic  surgery  has  transitioned 
through  a  similar  evolution  to  laparoscopic  surgery  and  is 
being  recognized  as  an  important  surgical  approach  by 
multiple  surgical  specialties.  Furthermore,  it  shows  every 
sign  of  continuing  the  adoption  of  more  diverse  surgical 
procedures,  as  manifest  by  the  fact  that  in  calendar  year 
2011,  approximately  350,000  robotic  surgical  procedures 
were  performed  (Figure  1).  The  number  of  procedures 
being  performed  by  robotic  surgery  has  been  constantly 
rising  in  urology,  gynecology,  colorectal,  pediatric  and 
numerous  other  specialties.  Expert  robotic  surgeons  and 
numerous  surgical  societies  and  certifying  organizations 
have  advocated  the  need  for  the  creation  of  a  unified 
approach  and  standardized  curriculum  for  basic  training 


and  certification  in  robotic  surgery  skills  [3].  There  have 
been  efforts  to  develop  a  core  curriculum  for  certifying 
robotic  surgeons  [4,5];  however,  these  have  been 
fragmented,  with  different  approaches  and  outcomes 
measures  emerging  from  each.  This  has  resulted  in 
conflicting,  competing  and  redundant  curricula  for  the 
training  and  the  assessment  tools  for  robotic  surgery.  In 
addition,  these  curricula  have  generally  lacked  the  human 
and  financial  resources  necessary  to  complete  the  most 
comprehensive,  multi -institutional  validation  that  is 
necessary  to  gain  acceptance  at  a  national  level. 

Through  the  combined  support  of  two  grants,  one  to  the 
Minimally  Invasive  Robotics  Association  and  the  other  to 
Florida  Hospital  Nicholson  Center,  we  have  created  a 
process  and  a  group  of  participants  which  unify  the 
previous  attempts  to  develop  a  robotic  curriculum  and 
expand  to  a  much  larger  foundation  of  surgical  societies 
with  a  stake  in  this  new  technology.  These  grants  provide 
the  necessary  funding  to  carry  the  effort  through  multi- 
institutional  validation  with  the  support  of  participants 
who  represent  all  surgical  specialties  that  are  currently 
performing  robotic  surgery. 

Methods  &  Materials 

Participation  in  this  effort  was  invited  from  multiple 
certifying  boards,  professional  surgical  societies,  and 
associations  that  represent  international  practitioners  and 
regulators  of  various  surgical  specialties  as  well  as  the 
United  States  Department  of  Defense  (DoD)  and  Veterans 
Health  Administration  (VHA)  (Table  1).  The  conference 
participants  are  members  of  these  organizations  or 
agencies  and  are  selected  to  be  able  to  provide  insight  into 
the  needs  of  their  organizations,  but  they  do  not  represent 
an  endorsement  or  acceptance  of  the  results,  and 
participation  does  not  imply  acceptance  by  the  societies, 
boards  or  agencies.  However,  the  AUA,  AAGL,  and 
SAGES  elected  to  appoint  and  send  representatives  who 
could  officially  speak  for  their  organizations’  needs  for  a 
robotic  curriculum  and  officially  accept  the  results  of  the 
consensus  conferences.  This  project  is  an  effort  to  provide 
the  stakeholders  with  the  best  scientific  evidence  upon 
which  to  base  their  decisions  regarding  implementation  of 
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a  fundamental  curriculum  to  meet  their  needs  while 
reducing  redundancy,  competition  and  duplication  of 
effort. 

Table  1.  Invited  Organizational  Representation  in 
Fundamentals  of  Robotic  Surgery. 

American  Association  Gynecologic  Laparoscopy  (AAGL)  * 
American  College  of  Surgeons  (ACS) 

American  Congress  of  Obstetrics  and-Gynecology  (ACOG) 
American  Urologic  Association  (AUA)  * 

American  Academy  of  Orthopedic  Surgeons  (AAOA) 

American  Association  of  Thoracic  Surgeons  (AATS) 

American  Association  of  Colo-rectal  Surgeons  (ASCRS) 
Minimally  Invasive  Robotic  Association  (MIRA) t 
Society  for  Robotic  Surgery  (SRS) 

Society  of  American  Gastrointestinal  and  Endoscopic  Surgeons 
(SAGES)  * 

American  Board  of  Surgery  (ABS) 

Accreditation  Council  of  Graduate  Medical  Education 
(ACGME) 

Association  of  Surgical  Educators  (ASE) 

Residency  Review  Committee  (RRC)  -  Surgery 
Royal  College  of  Surgeons-Ireland  (RCSI) 

Royal  College  of  Surgeons-London  (RCSL) 

Royal  College  of  Surgeons- Australia  (RCSA) 

U.S.  Department  of  Defense  (DoD)  * 

U.S.  Department  of  Veterans  Health  Affairs  (VHA) 

*  :  Official  Representative  Participation 
Funding  organizations. 


Each  consensus  conference  was  conducted  over  a  two-day 
period  using  a  modified  Delphi  method  [6].  This 
methodology  consisted  of  a  facilitator  who  captured  the 
input  and  guidance  of  the  participants.  This  input  was 
then  analyzed  for  common  concepts  to  create  a  list  of 
critical  items  in  robotic  surgery.  Previously  published 
material  from  a  single  institution’s  curriculum  was  used 
as  a  template  for  initial  idea  generation  [7,8].  The 
individual  outcomes  measures  and  curriculum  materials 
were  itemized  and  votes  taken  on  their  importance 
according  to  each  participant.  This  method  led  to  a 
composite  ranking  which  was  captured  in  a  draft  report. 
The  report  containing  the  first  group  ratings  was  then  sent 
to  each  participant  for  their  private  deliberation.  Each 
participant  then  submitted  a  second  set  of  scores  which 
were  informed  by  the  first  composite  scores,  but 
anonymous  to  other  group  members.  This  modified 
Delphi  Method  led  to  a  higher  level  of  consensus  around 
the  measures  and  the  curriculum.  It  also  identified  those 
items  for  which  there  was  little  group  support.  Those 
items  were  removed  from  the  list  of  outcomes  measures 
and  from  the  outline  of  the  curriculum. 

The  first  conference  on  outcomes  measures  was  attended 
by  20  participants  that  included  surgeons,  scientists, 
educators,  and  facilitators.  The  ranking  of  the  tasks 
identified  was  done  by  a  subset  of  nine  experienced 
surgeons.  Participants  who  were  not  surgeons  abstained 
from  the  scoring  process. 


The  second  conference  on  curriculum  development  was 
attended  by  38  surgeons,  scientists,  educators,  and 
facilitators.  This  group  reviewed  and  became  familiar 
with  the  material  from  the  first  conference.  Thereupon, 
they  were  divided  into  three  working  groups  to  develop 
curriculum  that  focused  on  didactic  and  knowledge-based 
information,  psychomotor  skills,  and  team  training  and 
communications.  Similarly,  the  actual  ranking  of  the 
material  developed  was  limited  to  experienced  surgeons 
within  the  group. 

Results 

The  first  consensus  conference  resulted  in  a  list  of  25 
outcomes  measures  which  the  group  agreed  should  be 
mastered  by  a  surgeon  seeking  privileges  in  robotics. 
These  included  8  pre-operative,  15  intra-operative  and  2 
post-operative  tasks  which  are  shown  in  Figure  2.  The 
resulting  report  also  provides  detailed  definitions, 
descriptions,  errors,  outcomes  and  metrics  for  each  of 
these  tasks  [9]. 

FRS  Outcomes  Measures 


Pre-Operative  |  Intra-Operative  |  Post-Operative 


System  Settings  Energy  Sources  Transition  to  Bedside  Asst 

Ergonomic  Positioning  Camera  Control  Undocking 

Docking  Clutching 

Robotic  Trocars  Instrument  Exchange 

OR  Set-up  Foreign  Body  Management 

Situation  Awareness  Multi-arm  Control 

Closed  Loop  Comms  Eye-hand  Instrument  Coord 

Respond  to  System  Errors  Wrist  Articulation 

Atraumatic  Tissue  Handling 
Dissection  -  Fine  &  Blunt 
Cutting 

Needle  Driving 
Suture  Handling 
Knot  Tying 

Safety  of  Operative  Field 

Figure  2.  FRS  Outcomes  Measures. 

The  second  consensus  conference  on  curriculum 
development  resulted  in  outlines  and  principles  for  the 
creation  of  a  curriculum  to  teach  the  previously  identified 
list  of  tasks  and  knowledge  (Figure  3). 

Didactic  and  Knowledge.  The  didactic  and  knowledge 
working  group  created  an  outline  of  the  material  which 
should  be  taught  in  lecture  format.  This  will  include: 

1 .  Introduction  to  robotic  surgical  devices. 

2.  Pre -operative  set-up  of  equipment  and 
positioning  of  staff. 

3.  Intra-operative  use  of  a  robot,  surgeon 
ergonomics,  visual  field  control,  and  necessary 
instruments  and  supplies. 

4.  Post-operative  steps  for  removing  a  robot  and 
transitioning  to  bedside  control. 

Each  of  these  included  an  explicit  list  of  errors  that  can 
occur  in  the  process. 
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FRS  Curriculum  Outline 


Didactic  &  Cognitive 

Psychomotor  Skills 

Team  Training 

Lecture-based 

Principle-based 

Checklist-based 

Intro  to  Robotic  System 

Based  on  Physical  Models 
(Virtual  Models  are  Derivative) 

#1:  WHO  Pre-Op 

Pre-Operative  Activity 

3D  Exam  Tools 

#2:  Robotic  Specific 

Intra-Operative  Activity 

Use  Tasks  that  have 

Evidence  of  Validity 

#3:  Undocking  & 
Debriefing 

Post-Operative  Activity 

Multiple  Outcomes 
Measured  per  Exercise 

#4  Crisis  Scenarios 

Each  Activity  includes: 

Goals,  Conditions,  Metrics, 
Errors,  Standards 

Cost  Effective  Solution 

High  Fidelity  for  Testing, 

Lower  Fidelity  for  Training 

IRR  Requires  Ease  of 

Administration 

Figure  3.  FRS  Curriculum  Outline  and  Principles. 


Psychomotor.  The  psychomotor  skills  working  group 
prefaced  their  work  with  seven  principles  that  should  be 
applied  in  selecting  or  designing  a  skills  device  for 
robotic  surgery.  Those  principles  were: 

1.  The  tasks  should  be  3  dimensional  in  nature. 

2.  The  tasks  designed  for  testing  should  be  such 
that  they  have  multiple  learning  objectives  that 
incorporate  multiple  tasks  from  the  first 
conference  report.  The  tasks  designed  for 
training  will  have  more  focused  learning 
objectives. 

3.  Implementation  of  the  tasks  and  the  resultant 
method  for  teaching  should  be  cost  effective. 

4.  High  fidelity  models  should  be  used  for  testing. 
Training  can  use  lower  fidelity  devices  or 
methods. 

5.  Tasks  should  be  easy  to  administer  to  ensure 
Inter-Rater  Reliability  (IRR). 

6.  The  tasks  should  be  designed  for  implementation 
with  physical  objects  and  devices.  Future 
implementation  in  VR  with  a  simulator  would  be 
derivative  of  the  physical  model. 

7.  Preference  should  be  given  to  tasks  that  have 
existing  evidence  of  validity 

The  group  then  identified  16  of  the  25  tasks  which 
contained  psychomotor  features.  To  address  these,  they 
proposed  ten  tasks  which  could  be  used  to  measure  these 
skills.  Three  tasks  were  drawn  from  FLS,  others  were 
selected  from  existing  educational  programs,  and  designs 
for  new  task  devices  were  proposed. 

1.  FLS  peg  transfer 

2.  FLS  suturing 

3.  FLS  pattern  cutting 

4.  Running  Suture 

5.  Dome  with  four  towers 

6.  Vessel  dissection  and  clipping 

7.  UTSW  4th  arm  retraction  and  cutting 

8.  Energy  and  mechanical  cutting 

9.  Docking  task  (new  design) 

10.  Trocar  insertion  task  (new  design) 


For  each  of  these  the  group  also  identified  the  associated 
task  description,  conditions,  metrics,  and  errors. 

Team  Training  and  Communications.  The  team  training 
and  communications  working  group  prefaced  their  work 
by  defining  the  importance  of  team  training  in  a  robotic 
environment.  They  identified  the  following  principles  as 
essential  to  successful  team-based  operations  and  training. 

1 .  Inclusion 

2.  Empowerment 

3.  Person  specific 

4.  Reiterative 

5.  ‘Just  in  time’ 

6.  Ownership 

7.  Risk  management/quality  improvement-  closed 
loop 

They  stated  that  existing  programs  like  TeamSTEPPS  can 
be  applied  to  robotic  teams.  Their  curriculum  follows  a 
checklist  format  and  is  conceptually  derived  from  the 
standard  WHO  checklist.  For  robotic  training  they 
recommended  the  following  checklists: 

1.  Pre -operative.  Addressing  General  situation, 
surgeon,  anesthetist,  nurse/OPD,  and  surgical 
site  infection. 

2.  Robotic  Docking.  Addressing  anesthesia,  patient, 
bedside  assist,  procedure-specific  checks,  and 
trouble  shooting. 

3.  Intra-operative.  Addressing  the  communication 
that  occurs  within  a  team  throughout  the 
operation. 

4.  Undocking  and  Debriefing. 

A  third  consensus  conference  is  scheduled  for  August 
2012  to  write  the  detailed  material  that  will  be  included  in 
the  didactic  and  team  training  sections  of  the  curriculum; 
and  where  specific  psychomotor  skills  devices  will  be 
identified,  designed  and  selected. 

Conclusions  &  Discussion 

Two  consensus  conference  involving  members  from 
major  stakeholder  organizations  in  surgical  training, 
governance,  and  certification  across  multiple  specialties 
have  been  conducted  to  arrive  at  a  consensus  regarding 
the  most  important  outcome  measures  for  the  safe  conduct 
of  robotic  surgery  and  the  curriculum  to  teach  those  skills 
and  knowledge.  The  development  of  FRS  is  multi¬ 
specialty,  system  agnostic  and  follows  decades  of 
experience  in  other  industries  at  developing  such 
education  and  training  platforms.  Using  the  curriculum 
for  training  and  assessment  should  result  in  a  surgeon 
who  has  proficiency  in  basic  robotic  surgery  skills  and  is 
capable  of  passing  the  requirements  of  high  stakes  testing 
and  evaluation.  At  some  future  time,  this  testing  and 
evaluation  would  be  administered  by  an  appropriate 
independent,  objective  and  authoritative  organization, 
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which  would  adopt  the  materials  developed  through  this 
consensus  process. 
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Robotic  and  Telesurgery  Research  Summary 


Telesurgery 


Comms  Latency: 

•Modify  surgical  procedures 
•Safe  Telesurgery  at  500ms 
•Match  to  City-Pairs 


Automatic  Surgery: 

•Record  Surgery  in  Simulator 
•Execute  with  Unmanned  Robot 
•Identify  Control  Variables 


Simulation 


Surgical  Rehearsal: 

•Dynamic  Organ  Model  in  Sim 
•Patient-specific  Rehearsal 
•Improve  Surgeon  Performance 


Military-use  Validation: 

•Simulator  of  Robotic  Surgery 
•Retain  Skills  in  Theater 
•Define  Deployable  Package 


Robotic  Curriculum 


Consensus  Conferences: 

•Define  Certification  Criteria 
•Develop  Curriculum 
•Develop  Training  Tasks 


Curriculum  Validation: 

•Validate  the  Program 
•Identify  Testing  Measures 
•Set  Passing  Criteria 


Telesurgery:  Communication  Latency 


•Stereo  HD  Video 
•Team  Audio 


•Robot  Commands 
•Surgeon  Audio 


Comm  Latency  = 


Telesurgery:  Simulate 


da  Vinci  Skills  Simulator 


Mimic  dV-Trainer 


Telesurgery:  Latency  Tolerance  (Concept) 
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Surgical  Procedure  Time 


Latency  Map:  City 


Potential  City  Pairs: 

Orlando,  FL 
Bethesda,  MD 
Seattle,  WA 
Boston,  MA 
New  York,  NY 
Atlanta,  GA 
Dallas,  TX 
Denver,  CO 
San  Fran,  CA 

Strasbourg,  FR 
Sao  Paulo,  BZ 
Tel  Aviv,  IS 


Telesurgery  Modifications 


Control  pace  of  movement 
Subdivide  current  atomic  movements 
Change  direction  of  movements 
Introduce  new  instruments 
Stabilize  tissue 


TOT 
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Simulation:  Surgical  Rehearsal 


Telesurgery:  Automatic  Surgery 


Pierce  the  object  at  the  indicated  location  and  pull  the 
redle  through.  Start  from  the  yellow  aide  of  the  target 


Simulation:  Military-use  Validation 
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Fundamentals  of  Robotic  Surgery 
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Future  Directions 


*  Robotics 


-  Machine  assistance  for  all  surgical  procedures.  “Robot"  will  take 
multiple  forms  to  fit  the  needs  of  the  procedure. 

-  Redesigning  the  operating  room  to  accommodate  people,  machines, 
and  information. 


*  Simulation 

-  Lap  and  Robotics  use  equipment  to  intermediate  between  the  surgeon 
and  the  patient.  Creates  a  natural  environment  for  training  simulators 

-  VR/Games/Browser  in  providing  in-hospital  maintenance  training. 
Currently  done  largely  with  in-service  seminars. 

*  Education 

-  Curriculum  that  integrates  lecture,  live,  and  simulation.  Nursing  has 
taken  the  lead  in  this,  surgery  catching  up. 
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Richard  M  Satava,  Sanket  Chauhan,  Roger  D  Smith,  Robert  M  Sweet,  Daniel  Scott,  Arnold 
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Kunkler,  COL  Robert  Rush,  Wallace  Judd,  Mika  Sinanan,  Dimitrios  Stefanidis,  Chandru 
Sundaram,  Edward  Verrier,  Vipul  R  Patel 


Abstract 

Background:  Robotic  surgery  has  diffused  into  clinical  surgical  practice  and 
provides  a  minimally  invasive  opportunity  for  several  surgical  procedures  across 
multiple  specialties.  However,  no  comprehensive  basic  skills  curriculum  has  been 
validated  to  ensure  the  proficiency  of  the  surgeon  who  is  using  the  robot.  A 
rigorous,  standard  methodology  is  being  developed  to  create  a  basic  robotic 
surgery  skills  curriculum  called  the  Fundamentals  of  Robotic  Surgery  (FRS].  The 
purpose  of  this  report  is  to  describe  the  results  of  the  first  consensus  conference  - 
Outcome  Measures -that  will  serve  as  the  foundation  for  the  full  FRS  curriculum. 
Methods:  Surgeons  from  19  different  surgical  societies,  regulatory  bodies  and/or 
federal  agencies  across  multiple  specialties  were  invited  to  participate  in  the 
consensus  conference.  Among  the  participants  were  nine  practicing  clinical 
robotic  surgeons,  7  professional  surgical  educators,  8  executive  members  of 
surgical  societies  or  boards,  3  representatives  from  DoD,  and  1  representative  of 
the  federal  government’s  National  Institute  for  Standards  and  Testing.  Using  a 
standard  task-deconstruction  methodology,  the  group  identified  the  basic 
tasks  and  their  outcome  measures  required  for  robotic  surgery.  Task 
importance  was  rated  using  a  modified  Delphi  methodology  to  arrive  at 
group  consensus  and  tasks  with  a  score  below  2  standard  deviations  from 
the  mean  were  excluded.  Results:  A  total  of  26  tasks  were  identified  (8  pre¬ 
operative,  15  intra-operative  and  3  post-operative ).  After  the  second  round  of 
Delphi  voting,  the  mean  score  was  28.19  (range:  18-35;  SD:  4.94).  Situational 
awareness  received  the  highest  score  (35)  followed  by  eye  hand  instrument 
coordination,  needle  driving  and  atraumatic  handling  (33).  Transition  to  bedside 
(20)  and  clip  applying  (18)  received  the  minimum  score.  The  threshold  score  was 
19.77  and  following  the  final  Delphi  round,  one  task  (Clip  Applying)  was  excluded 
from  the  final  task  list. 

Conclusions:  The  first  FRS  consensus  conference  for  Outcomes  Measures  was 
conducted  to  identify  the  candidate  tasks  and  outcome  measures  for  the  FRS 
curriculum  development.  The  final  25  tasks  will  be  used  in  the  next  consensus 
conference  on  Curriculum  Development  to  guide  the  development  of  a  standard 
curriculum.  This  will  be  followed  by  a  third  consensus  conference  to  design  a 
Validation  Study. 
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Introduction 

The  introduction  and  subsequent  adoption  of  any  new  technology  into  clinical  practice  opens 
further  frontiers  for  more  effective  management  or  even  a  potential  cure  for  several  diseases.  However, 
with  the  exponential  growth  of  technology  in  the  Information  Age,  the  training  and  certification  of 
surgeons  to  perform  a  new  procedure  safely  and  effectively  poses  a  unique  challenge.  For  example,  the 
proliferation  of  laparoscopic  surgery  as  the  new  gold  standard  for  symptomatic  cholelithiasis  initially  led 
to  an  increase  in  common  bile  duct  injuries  [1-3] .This  rightfully  cautioned  the  governing  bodies, 
academic  surgical  community  and  even  surgeons  at  large  to  develop  methods  that  would  ensure  that 
only  competently  trained  surgeons  perform  such  complicated  procedures,  which  frequently  required  a 
different  skill  set  than  open  procedures. 

In  2005,  the  Accreditation  Council  for  Graduate  Medical  Education  (ACGME)  recognized  the 
importance  of  using  simulation  in  the  training  and  assessment  of  surgical  technical  skills  and  required 
that  by  2008,  surgical  residency  programs  have  access  to  a  simulation  center  [4],  In  2004,  the  Society  of 
American  Gastrointestinal  and  Endoscopic  Surgeons  (SAGES)  launched  the  validated  Fundamentals  of 
Laparoscopic  Surgery  (FLS)  curriculum  and,  together  with  the  American  College  of  Surgeons  (ACS), 
promoted  the  FLS  as  a  minimum  standard  before  a  surgeon  should  be  allowed  to  perform  laparoscopic 
procedures  independently  [5],  In  2009,  The  American  Board  of  Surgery  (ABS)  mandated  that  in  addition 
to  Advanced  Cardiac  Life  Support  (ACLS)  and  Advanced  Trauma  Life  Support  (ATLS)  a  certificate 
documenting  the  successful  passing  of  the  FLS  exam  be  included  in  the  application  in  order  to  be  eligible 
to  sit  the  examination  for  certification  in  General  Surgery  [6], 

During  the  last  decade,  robotic  surgery  has  transitioned  through  a  similar  evolution  as 
laparoscopic  surgery  and  is  being  recognized  as  an  important  surgical  approach  by  multiple  surgical 
specialties.  Furthermore,  it  shows  every  sign  of  continuing  the  adoption  of  more  diverse  surgical 
procedures,  as  manifest  by  the  fact  that  in  calendar  year  2011,  approximately  350,000  robotic  surgical 
procedures  were  performed  (Figure  1).  The  number  of  procedures  being  performed  by  robotic  surgery 
(a  standard  in  neurosurgery  and  urologic  surgery)  has  been  constantly  rising  not  only  in  urology,  but 
gynecology,  colorectal,  pediatric  and  numerous  other  specialties  -  hence  the  inclusion  of  the  many 
surgical  specialties.  Expert  robotic  surgeons  and  numerous  surgical  societies  and  certifying 
organizations  have  advocated  the  need  for  the  creation  of  a  unified  approach  and  standardized 
curriculum  for  basic  training  and  certification  in  robotic  surgery  skills  [7],  Unfortunately,  the  excellent 
results  of  the  adoption  of  FLS  has  no  documentation  as  to  the  method  of  developing  the  curriculum 
which  could  be  adapted  for  the  FRS,  therefore  this  manuscript  includes  a  detailed  methodology  for 
curriculum  development  which  is  intended  to  not  only  support  the  FRS,  but  subsequent  robotic  and 
non-robotic  curricula  which  will  be  developed  within  the  specialties,  providing  a  common  methodology 
so  each  time  a  curriculum  is  developed  it  is  not  necessary  to  'reinvent  the  wheel'.  In  addition,  there  have 
been  current  efforts  to  develop  a  core  curriculum  for  certifying  robotic  surgeons  [8,9];  however,  this  is  a 
fragmented  effort,  with  different  approaches  and  outcomes  measures.  This  has  resulted  in  conflicting, 
competing  and  redundant  curricula  for  the  training  and  the  assessment  tools  for  robotic  surgery.  In 
addition,  these  curricula  have  generally  lacked  the  financial  resources  necessary  to  complete  the  most 
comprehensive,  multi-institutional  validation  that  is  necessary  to  gain  acceptance  at  a  national  level. 

The  FRS  initiative  is  a  collaboration  between  the  academic  community  and  the  federal 
government  to  assist  in  developing  an  interoperable,  uniform  curriculum  that  is  acceptable  across 
multiple  surgical  specialties  (Table  1).  A  major  premise  is  that  this  curriculum  will  only  address  the  most 
basic  of  skills  necessary  to  safely  conduct  robotic  surgery  and  that  the  participating  surgical  societies  will 
develop  respective  fundamental  robotic  surgery  curricula  that  are  still  very  basic  but  unique  to  their 
specialty,  such  as  the  Fundamentals  of  Urologic  Robotic  Surgery  or  the  Fundamentals  of  Gynecologic 
Robotic  Surgery.  An  important  concept  is  that  the  curriculum  template  that  is  used  to  craft  the  FRS  can 
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form  an  interoperable  platform  for  developing,  in  a  modular  fashion,  the  subsequent  curricula  that  are 
needed  for  each  specialty.  This  concept,  was  first  presented  by  Dr.  Robert  Sweet,  Department  of 
Urology,  University  of  Minnesota  in  2010  during  an  ACS  Accreditation  Educational  Institute  conference 
and  has  been  referred  to  as  the  "Sweet  T ree"  of  curriculum  development  (Figure  2). 
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Figure  1  Increase  in  number  of  robotic  surgical  procedures 
Source:  Intuitive  Surgical,  Inc  Investor  Prospectus,  Feb,  2012 
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Table  1:  FRS  Mission  statement,  specific  goals  and  deliverables  of  the  consensus  conferences. 


Mission  statement: 

"To  create,  develop  and  validate  a  multi-specialty,  technical  skills  and  competency  based  curriculum  for 

surgeons  to  safely  and  efficiently  perform  basic  robot-assisted  surgery." 

Goals: 

1.  Identify  the  outcomes  measures,  which  will  be  required  by  the  curriculum  and  the  methods  of 
measuring  them. 

2.  Integrate  how  and  which  process  will  be  used  to  acquire  the  outcomes  measures. 

3.  Define  the  tasks/subtasks  that  are  to  be  measured  using  task  analysis  methods. 

4.  Match  the  list  of  the  specific  tasks/subtasks  with  the  desired  outcomes  measurements  and  how 
those  measurements  will  be  acquired. 

Deliverables: 

The  report  of  the  outcomes  measures  meeting,  to  be  made  available  for  the  next  curriculum  meeting  of 

FRS,  which  specifically  lists: 

1.  Appropriate  outcomes  needed  to  train,  assess,  and  certify  the  most  fundamental  skills  in  robotic 
surgery 

2.  The  suggested/preferred  methods  of  measuring/acquiring  the  metrics 

3.  The  suggested  tasks/subtasks  for  the  FRS 

4.  The  actual  quantitative/qualitative  measures  that  need  to  be  measured  for  the  individual 
tasks/subtasks  that  comprise  the  FRS 
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Development  of  Curriculum  from  common  template 

“Sweet*  Tree” 


*  Adapted  from  Rob  Sweet,  MD,  Professor  of  Urology,  University  Minnesota,  2010 

Figure  2.  The  "Sweet  Tree"  of  curriculum  development 


The  curriculum  for  FRS  is  focused  on  multidisciplinary  shared  skills  at  the  "trunk"  of  the  tree.  It  is 
being  designed  to  be  open  source,  based  upon  the  integrated  effort  of  the  various  key  stakeholders' 
stated  needs.  This  will  be  validated  in  a  multi-institutional  validation  study  that  will  be  scientifically 
stringent  enough  to  meet  the  criteria  of  high  stakes  testing  and  evaluation  such  as  board  certification 
and  recertification  for  performing  robotic  surgery.  This  will  be  available  to  societies  and  certification 
authorities  across  multiple  specialties.  The  full  life  cycle  of  the  FRS  curriculum  will  be  developed  through 
a  series  of  at  least  three  consensus  conferences:  (1)  Outcomes  measures,  (2)  Curriculum  development 
and  (3)  Validation  design,  (as  illustrated  in  Figure  3)  using  a  "backward  design"  methodology  (analogous 
to  reverse  engineering)  that  has  been  developed  and  iterated  by  the  non-medical  education  and 
simulation  community  over  the  past  80  years  [10].  The  purpose  of  this  manuscript  is  to  report  the 
results  of  the  first  consensus  conference  in  the  series,  which  was  held  on  December  12th  and  13th,  2011 
at  the  Florida  Hospital  Nicholson  Center  (FHNC). 
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The  Metrics  Drives  the  Process 
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Figure  3.  The  metrics  drive  the  process:  Full  life  cycle  development  of  a  curriculum 


Material  and  Methods: 

Participating  organizations:  Invitations  to  participate  in  the  consensus  conferences  were  sent  to 
a  wide  variety  of  stakeholders  which  included  multiple  certifying  boards,  professional  surgical  societies, 
and  associations  that  represent  practitioners  and  regulators  of  various  surgical  specialties  and  the 
United  States  DoD  (Table  2).  While  the  conference  participants  are  members  of  these  organizations  or 
agencies  and  are  selected  to  be  able  to  provide  insight  into  the  needs  of  their  organization,  they  do  not 
represent  any  endorsement  or  acceptance  of  the  results,  and  participation  does  not  imply  explicitly  or 
implicitly,  any  level  of  acceptance  by  the  societies,  boards  or  agencies.  However,  the  AUA  and  AAGL 
elected  to  appoint  and  send  official  representatives  to  speak  for  their  organizations'  needs  for  a  robotic 
curriculum  and  officially  accept  the  results  of  the  consensus.  It  is  the  intent  however  that  many 
appropriate  stakeholders  would  be  willing  to  review  the  results  of  the  final  validated  curricula  and 
choose  to  implement  it  in  a  manner  that  would  be  acceptable  to  their  standards  and  principles.  This 
process  is  a  grass  roots  effort  to  provide  the  stakeholders  with  the  best  scientific  evidence  upon  which 
to  base  their  decisions  regarding  implementation  of  a  fundamental  curriculum  that  can  help  create  a 
uniform  approach  to  curriculum  development  that  is  efficient,  cost  effective  and  flexible  enough  to 
meet  their  needs  while  reducing  redundancy,  competition  and  duplication  of  effort. 
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Table  2:  Organizations  invited  to  send  representatives. 
American  Association  Gynecologic  Laparoscopy  (AAGL)  * 

American  College  of  Surgeons  (ACS) 

American  Congress  of  Obstetrics  and-Gynecology  (ACOG) 

American  Urologic  Association  (AUA)  * 

American  Academy  of  Orthopedic  Surgeons  (AAOA) 

American  Association  of  Thoracic  Surgeons  (AATS) 

American  Association  of  Colo-rectal  Surgeons  (ASCRS) 

Department  of  Defense  Central  Simulation  Committee  (CSC) 

Minimally  Invasive  Robotic  Association  (MIRA) + 

Society  for  Robotic  Surgery  (SRS) 

Society  of  American  Gastrointestinal  and  Endoscopic  Surgeons  (SAGES) 
American  Board  of  Surgery  (ABS) 

Accreditation  Council  of  Graduate  Medical  Education  (ACGME) 
Association  of  Surgical  Educators  (ASE) 

Residency  Review  Committee  (RRC)  -  Surgery 
Royal  College  of  Surgeons-lreland  (RCSI) 

Royal  College  of  Surgeons-London  (RCSL) 

U.S.  Department  of  Defense  (DoD) + 

U.S.  Department  of  Veterans  Health  Affairs  (VHA) 

*  :  Official  Representative  Participation 
f :  Funding  organizations. 


The  modified  Delphi  method  The  Outcomes  Measures  Workshop  was  conducted  over  a  two- 
day  period  using  a  modified  Delphi  method  [11],  This  typical  methodology  consists  of  a  facilitator  who 
works  with  the  participants  in  large  sessions  and  smaller  break-out  groups  to  capture  the  ideas  and 
suggestions  generated  by  the  participants.  For  our  conference,  the  ideas  were  then  analyzed  for 
common  concepts  to  produce  a  list  of  critical  items  in  robotic  surgery.  A  previously  published  tasks 
deconstruction  list  from  a  single  institution's  curriculum  was  used  as  a  template  for  initial  idea 
generation  [12,  13],  The  individual  items  were  then  prioritized  by  value,  rank-order,  or  sequence  in  a 
preliminary  table,  graph  or  narrative  form.  From  each  session,  a  'reporter'  was  chosen  from  the  group 
to  summarize  and  deliver  the  results  of  the  session  to  the  group  as  a  whole  for  further  refinement.  The 
final  session  of  the  conference  was  a  review  of  the  consolidated  results  and  final  comments  and  critique 
by  the  participants.  Following  the  conference,  the  first  draft  of  the  report  was  generated  by  the 
principal  investigators  of  the  program  and  a  second  round  anonymous  Delphi  rating  was  used  to  achieve 
greater  concurrence,  to  prioritize  the  ranking  of  the  tasks,  and  to  eliminate  low-scoring  tasks. 

Scoring  method:  The  first  round  of  voting  was  carried  out  in  an  open  forum  with  participants 
indicating  their  opinion  on  the  importance  of  the  task  on  a  four  point  scale,  where  1  =  not  important,  2  = 
somewhat  important,  3  =  important,  and  4  =  critically  important.  The  votes  were  tallied  in  the  task 
matrix  and  a  total  score  was  computed  by  multiplying  the  scoring  level  by  the  number  of  votes  at  that 
level  and  summing  across  all  levels. 

Total  Score  =  ^  Rating .  *  NumVotesi 
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The  tasks  were  then  rank  ordered  based  on  each  of  their  Total  Scores  where  a  higher  score 
results  in  a  higher  order  in  the  ranking.  In  the  event  of  ties  in  the  total  score,  the  tasks  received 
sequential  places  in  the  ranking.  This  ranking  identified  the  level  of  support  that  existed  within  the 
group.  The  results  were  compiled  into  a  Task  Matrix  (Table  3)  which  identified  the  tasks,  definitions, 
error  conditions,  measurement  methods,  importance  scores  and  ranking  within  the  group. 

Table  3.  FRS  Task  List,  Ranked  Ordered  by  Total  Score,  Following  2nd  Round  Delphi  Scoring 


Task  Name 

Description 

Errors 

Outcomes 

Metrics 

Importance  Rating 

1 

2 

3 

4 

Total 

Score 

Rank 

Order 

Situation 

awareness 

Awareness  of 
the  status  and 
readiness  of  the 
people  and 
equipment 
essential  to  the 
operation. 

Unaware  of 
Robot-Patient- 
Assistant  -team 

state 

Maintain 
awareness  of  the 
robotic,  patient, 
and  team  status 
that  is  out  of 
view. 

Missed 

communication. 
Missed  information. 
Missed  changes  in 
patient  status  and 
injuries,  missed 
changes  in  robotic 
status 

0 

0 

1 

8 

35 

1 

Eye-hand 

instrument 

coordination 

Using  the 
manual 
controls  to 
accurately 
manipulate 
bedside 
instruments 
and  perform 
tasks.  Passing 
objects 
between  the 
instruments. 

Ineffective 

targeting 

Efficient  hand 
coordination 
and  accurate 
and  efficient 
movement  of 
instruments 

Time  and  economy 
of  motion 

0 

0 

3 

6 

33 

2 

Needle  driving 

Accurate  and 
efficient 
manipulation  of 
the  needle. 

Tearing  tissue, 
Troughing  the 
needle, 

Needle 

scratching, 

Wrong  angle  on 
entry/exit, 
Adjacent  organ 
injury, 

Needle  damage, 

Needle 

positioning, 

Needle 

dropping, 

Holding  out  of 
field-of-view, 
Poor  accuracy 

Accurate  and 
efficient 
placement  of 
needle  through 
targeted  tissue, 
Following  the 
curve  of  the 
needle,  without 
associated  tissue 
injury 

Time,  accuracy, 
tissue  damage, 
material  damage 

0 

0 

3 

6 

33 

3 

Atraumatic 

handling 

Haptic 

comprehension. 
Using  graspers 
to  hold  tissue 

Traumatic 

handling, 

Tissue  damage 
or  hemorrhage 

Manipulates 
tissue  and 
surgical 
materials 

Metric-respect  for 
tissue, 

Stress  and  strain 
indentation  and 

0 

0 

3 

6 

33 

4 

8 


134 


FRS  Consensus  Conference  I:  Outcomes  Measures 


Manuscript  Draft,  June  16,  2012 


Task  Name 

Description 

Errors 

Outcomes 

Metrics 

Importance  Rating 

1 

2 

3 

4 

Total 

Score 

Rank 

Order 

or  surgical 
material 
without 
crushing  or 
tearing. 

Respect  to 
tissue 

without  damage 

deformation 

Safety  of 
Operative 

Field 

Appropriate 
insertion  and 
positioning  of 
instruments. 

Instrument 
collision  with 
tissue  outside  of 
field-of-view 

Effectively 
avoids 
instrument 
collision  and 
damage  with 
tissue  outside  of 
field  of  view 

Instrument  to  tissue 
contact,  tissue 
damage 

0 

1 

2 

6 

32 

5 

Camera 

Maneuvering 
the  camera  to 
obtain  a 
suitable  view 

Not  focused, 
Wrong  distance 
to  tissue, 
Inappropriate 
field  of  view. 
Disorientation 

on  camera 

orientation. 
Inappropriate 
choice  of 
camera  angle, 
Camera  contact 
with  tissue 

Maintains 

optimal 

imaging, 

including 

horizontal 

orientation, 

field-of-view, 

angle  at  all 

times. 

Time, 

efficiency(clutching), 
sizing(magnification 
and  field  of  view), 
horizontal 
orientation,  camera 
tissue  contact, 
control  and 
manipulation, 
smoothness,  scope 
angle  selection 

0 

0 

4 

5 

32 

6 

Clutching 

Maintaining 
comfortable 
range  of 
motion  for 
manual 
controls 

[Extension  of 
Ergonomics] 

Loss  of  range- 
of-motion 

Efficiently 
maintains  full 
range  of  motion 
at  all  times,  in 
an  ergonomic 
manner. 

Joystick  collisions, 
joystick  maintained 
within  fly 

zone(establish  what 
fly  zone  is),  efficient 
control  system 
usage(excessive 
clutching,  wrong 
pedal) 

0 

1 

2 

6 

32 

7 

Dissection  - 
fine  &  blunt 
(Traction/ 
counter¬ 
traction) 

Using 

instruments  to 
perform  precise 
or  blunt 
dissection  of 
structures 

Failure  to 
identify  correct 
tissue  plane, 
Inadequate 
traction/counter- 
traction. 
Reversing  blunt 
vs.  fine 

Performs 
dissection  in 
appropriate 
planes  with 
suitable 

traction/counter¬ 
traction  and 
without 
collateral 
damage 

Accuracy  and 
damage  to 
surrounding 
structures, 
distribution  of  force 
across  tissue, 
time,&  provides 
adequate  exposure  of 
target  tissue 

0 

0 

4 

5 

32 

8 

Closed  loop 
communication 

Definitive 
communication 
techniques 
between  the 
members  of  the 

Communication 

failure, 

Incorrect 

terminology 

Actions  match 
intent  between 
team  members. 
Use  of  names, 
individual 

Use  of  names,  clarity 
of  request,  response 
time,  call  back 
requested  and 
provided 

0 

0 

5 

4 

31 

9 

9 
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Task  Name 

Description 

Errors 

Outcomes 

Metrics 

Importance  Rating 

1 

2 

3 

4 

Total 

Score 

Rank 

Order 

surgical  team. 

responsibilities 
given,  follow¬ 
up  information 
provided. 

(TeamSTEPPS®) 

Docking 

Surgeon  guides 
OR  nurse  in 
positioning 
bedside  robot 
and  attaches 

arms  to  trocars 

External 
collision. 
Misalignment, 
Bed  movement 
post-docking 

Appropriately 
docks  robot  in 
timely  fashion 
with  minimal 
adjustments. 

Time  to  dock, 
adjustment ,  patient 
or  instrument 
collision,  robotic  arm 
position,  alignment 

0 

1 

3 

5 

31 

10 

Knot  tying 

Exactness  of 
the  creation  of 
a  knot  with 

suture. 

Air  knot, 

Knot  slippage, 
Insecure  knot, 
Inappropriate 
tail  length, 

Bunny  ears, 

Too  tight. 

Tissue  ischemia 

Ties  secure 
knots 

appropriately, 
accurately  and 
efficiently 
without  tissue 
damage 

Time,  economy  of 
motion,  tissue 
damage,  material 
damage,  knot 
location,  air  knot, 
knot  security, 
protocol  violation, 
appropriate  tail 
length 

0 

1 

4 

4 

30 

11 

Instrument 

exchange 

Changing  out 
instruments 
used  in  the 
operation 

Tissue  collision 

during 

exchange, 

Non-visualized 

or  memory- 

guided 

instrument 

insertion. 

Inserting  or 
removing  the 
wrong 
instrument 

Efficient, 
accurate  and 
safe  instrument 
exchange 
without  tissue 
collision. 

Tissue  damage,  time, 
economy  of  motion, 
connection  to  energy 
source,  coordination 
with  assistant, 
instrument  selection, 
recognition  of 
instrument  failure, 
proper  instrument 
engagement  to 
robotic  arm  and  port, 
memory  recognition, 
trouble  shoot 
protocol 

0 

0 

7 

2 

29 

12 

Cutting 

Using  the 
scissors  to  cut 
at  a  precise 
location 

Cutting  the 
wrong  structure, 
Past-pointing, 
Inappropriate 
instruments 

Accurate  and 

efficient 

division  of 

target  structure 

without 

collateral 

damage 

Accuracy,  lack  of 
tissue  damage, 
timeliness 

0 

1 

5 

3 

29 

13 

Energy  sources 

Activation  and 
control  of 
cautery  or  other 
energy  sources 

Mirror  FLS 

errors, 

Pedal  to 

instrument 

discordance, 

Activate  energy 

before  tissue 

contact, 

Unintentional 

energy 

activation. 

Unintentional 

Appropriate 
choice  and  use 
of  energy 
sources  with  no 
collateral 
damage. 

Collateral  tissue 
damage  (real  time 
and  delay  contact), 
instrument  and 
energy  choice, 
activation  without 
tissue  contact, 
economy  of  energy 
use  (air  burns),  pedal 
selection 

0 

2 

4 

3 

28 

14 

10 
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Task  Name 

Description 

Errors 

Outcomes 

Metrics 

Importance  Rating 

1 

2 

3 

4 

Total 

Score 

Rank 

Order 

energy  arcing 

Foreign  body 
management 

Removal  of  all 
foreign  bodies 
from  the 
operating 
space. 

Failure  to 
confirm  foreign 
body  removal 
(needle,  sponge, 
bulldog) 

Safe, 

appropriate  and 
confirmed 
foreign  body 
removal 

Instrument  selection 
for  removal,  correct 
instrument,  sponge 
and  needle  count, 
removal  technique, 
immediate 
confirmation  of 
removal 

1 

1 

3 

4 

28 

15 

Robotic  trocars 

Safe  insertion 
technique. 

Incorrect  remote 
center, 

Trocar  slippage, 

Spatial 

orientation. 

Blind  insertion 
(2nd  and  later), 
Organ  injury, 
Access  will 
mirror  FLS 

Appropriate 
trocar  insertion 
and  positioning 
relative  to  target 
and  other 
trocars,  without 
unintentional 
tissue  contact. 
Maintaining 
positioning. 

Time,  tissue  damage, 
number  of 
adjustments,  remote 
center  placement, 
distance  between 
trocars,  trocar  spatial 
relationship  to  target 

0 

2 

4 

3 

28 

16 

Suture 

handling 

Running  and 
interrupted 
sutures 
(separate  or 
combined) 

Breaking  suture, 
Fraying  suture, 
Tissue  tearing, 
Inadequate 
following. 

Poor  tension, 

Inadequate 

tissue 

coaptation. 

Inadvertent 

locking 

Appropriate 
handling  of 
suture  material 
without  fraying, 
breakage,  or 
tissue  damage. 

Tissue  damage,  time, 
accuracy,  economy 
of  motion,  material 
damage 

0 

2 

5 

2 

27 

17 

Wrist 

articulation 

Understanding 
and  utilizing 
the  full  range 
of  motion  of 
the  endowrist 

Not  using  all 
degrees-of- 
freedom, 
Inadvertent 
trapping  of 
tissue  or  suture 

Uses  all  degrees 
of  freedom 
appropriately 

Time,  dexterity  and 
economy  of  motion 

0 

2 

5 

2 

27 

18 

Ergonomic 

positioning 

Positioning  of 
the  surgeons 
torso,  arms  and 
feet. 

Poor  posture, 

elbow 

placement 

Maintains 
appropriate 
posture  and 
ergonomics 
throughout  the 
operation  and 
minimizes 
fatigue. 

Work  load,  posture, 
muscle  fatigue 

0 

1 

8 

0 

26 

19 

System 

settings 

Setting  up  and 
adjusting 
console 
settings  as 
needed  during 

Improper 
console  settings, 
Scope  angle 
selections, 
Magnification 

Appropriate 
console  settings 
with  minimal 
ongoing 
adjustments 

Number  of 
adjustments,  correct 
console  settings, 
checks  settings,  time 

0 

2 

6 

1 

26 

20 

11 


137 


FRS  Consensus  Conference  I:  Outcomes  Measures 


Manuscript  Draft,  June  16,  2012 


Task  Name 

Description 

Errors 

Outcomes 

Metrics 

Importance  Rating 

1 

2 

3 

4 

Total 

Score 

Rank 

Order 

surgery 

setup, 

Motion 

speed/scaling 

Multi-arm 

control 

Activating  the 
fourth  arm 
through 
clutching  and 
using  it  in  the 
operation 

Collision, 

Moving  wrong 
arm 

Efficient  use  of 
multi-arm 
control  without 
collisions 

Time  and  number  of 
collisions 

0 

3 

4 

2 

26 

21 

Operating 

Room  (OR) 
set-up 

Placing  the 
bedside  cart  in 
the  location 
where  the 
operative  field 
is  most 
accessible 

Incorrect 

support 

equipment 

placement. 

Breaking  sterile 

field 

Proper 
placement  of 
equipment  in  a 
sterile  and  safe 
fashion 

Breaks  in  sterile 
protocol,  equipment 
placement,  access 
and  visualization  for 
assistant,  time,  time 
to  conversion, 
access/clearance  to 
patient  cart  for  rapid 
undocking 

0 

3 

6 

0 

24 

22 

Respond  to 
robot  system 
error 

Understand  the 

robotic 

protocol 

Protocol 

violation 

Identifies 

correct 

troubleshooting 
algorithm  and 
applies  steps  in 
a  timely  fashion 
to  correct  the 
error.  Avert 
unnecessary 
conversion. 

Protocol  violations, 
algorithm 
identification  and 
correct  response, 
time 

1 

2 

5 

1 

24 

23 

Undocking 

Removal  of 
robotic 
equipment 
from  the 
trocars  and 
patient 

Undocking 

without 

instrument 

removal. 

Tissue  damage 

Safe  and 
efficient 
undocking  of 
cart  in  routine 
and  emergency 
situations 

Time,  protocol 
violation,  tissue 
damage,  collisions 

0 

6 

2 

1 

22 

24 

Transition  to 
bedside  assist 

Instrument 

removal 

Tissue  damage, 
Lack  of  port-site 
inspection 

Safe  and 
efficient 
removal  of 
instruments  and 
ports 

Time,  inspection  of 
port  sites,  bleeding, 
tissue  damage 

0 

7 

2 

0 

20 

25 

Clip  applying 

Accurate 
application  of 
clips. 

Cross-clipping, 
Short-clipping, 
Poor  accuracy, 
Inadequate 
coaptation 

Places  the  clips 
accurately, 
appropriately 
and  securely 
without  crossing 
and  without 
leakage 

Time,  accuracy, 
crossed  clips,  clip 
damage,  incomplete 
and  ineffective  clip 
placement 

2 

5 

2 

0 

18 

26 

12 
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Second  round  Delphi  voting.  After  the  conclusion  of  the  conference,  the  task  matrix  was  edited, 
the  scores  compiled,  and  the  rankings  assigned.  The  compiled  results  were  emailed  to  the  members  of 
the  group  for  the  next  step  in  the  process.  Each  member  considered  the  first  round  scores  and  rankings 
of  the  tasks  in  private,  and  submitted  a  new  vote  on  the  importance  of  each  task  using  the  same  scoring 
scale.  The  second  vote  was  then  compiled  and  the  scores  were  examined  to  determine  whether  the 
second  Delphi  round  significantly  changed  the  total  score  and  ranking  for  each  task. 

Eliminated  tasks.  Tasks  that  received  a  total  score  more  than  two  standard  deviations  below  the 
mean  score  were  then  tagged  for  elimination  from  the  list.  These  tasks  will  not  be  included  in  the 
ongoing  development  of  a  curriculum  or  validation. 

Results.  Nineteen  participants  attended  the  two-day  conference  that  was  organized  at  FHNC.  The 
surgeons  within  the  group  had  been  in  clinical  practice  for  a  median  of  15  years  (IQR:  9-24  years),  all  of 
which  have  been  actively  involved  in  simulation  research,  administration,  or  training  for  a  median  of  10 
years  (IQR:  9-13.5  years).  Finally  nine  surgeons  who  actively  practiced  robotic  surgery  participated  in  the 
voting. 

A  total  of  26  tasks  were  identified  during  the  workshop  (8  pre-operative,  15  intra-operative  and 
3  post-operative).  Following  an  initial  group  discussion  the  definitions,  description,  errors,  outcomes 
and  metrics  for  each  task  were  defined. 

After  the  first  round  of  Delphi  voting,  situational  awareness,  eye-hand  instrument  coordination 
and  needle  driving  received  the  highest  score  of  32. These  were  followed  by  camera  maneuvering, 
clutching,  atraumatic  handling,  knot  tying  and  safety  of  operative  field  which  each  received  a  total  score 
of  31.  Following  the  face-to-face  conference,  a  final  round  of  voting  using  a  modified  Delphi  Method  was 
conducted  via  email.  The  voting  members  received  the  original  list  with  the  initial  rank  ordered  list  of 
the  tasks,  along  with  a  basic  statistical  analysis  of  the  distribution  of  those  scores.  They  were  asked  to 
vote  again  on  the  rating  of  each  task  to  see  if  the  knowledge  of  the  group  scores  and  the  associated 
ranking  of  the  tasks  would  cause  them  to  change  their  score.  After  the  second  round  of  scoring,  the 
mean  score  was  28.19  (range:  18-35;  SD:  4.94).  Situational  awareness  maintained  the  highest  score  of 
35,  followed  by  eye-hand  instrument  coordination,  needle  driving  and  atraumatic  handling  (33). 
Transition  to  bedside  assist  and  clip  applying  received  the  lowest  scores  of  20  and  18  respectively  (Table 
3).  The  threshold  score  to  be  included  in  the  outcomes  measures  (Mean  minus  two  standard  deviations) 
was  19.77.  Based  on  this  threshold  clip  applying  was  eliminated. 

Discussion.  The  purpose  of  this  workshop  on  Outcomes  Measures  was  to  define  the  outcomes  needed 
to  develop,  and  subsequently  to  be  used  to  validate,  the  FRS  curriculum.  The  recommendation  is  to  use 
the  rank-ordered  task  list  (Tablel)  as  the  outcomes  measures  which  are  judged  as  the  most  critical  in 
defining  competency  in  robotic  surgery  skills,  within  the  context  of  patient  safety.  The  list  in  Table  3 
contains  the  outcomes  measures  that  will  drive  the  second  conference  on  Curriculum  Development. 
The  list  identifies  the  most  basic  tasks  needed  to  develop  a  FRS  curriculum  to  teach  the  skills  necessary 
to  use  a  robotic  surgery  platform  safely,  regardless  of  surgical  specialty.  While  it  is  difficult  to  predict 
what  new  surgical  robotic  platforms  will  emerge,  the  outcomes  measures  from  this  conference,  along 
with  the  subsequent  curriculum,  will  be  the  foundation  upon  which  the  cognitive,  psychomotor  and 
team  training  skills  will  be  tested  for  robotic  surgical  platforms. 

Previous  curricula  on  technical  skills  have  had  the  shortcoming  of  not  being  adopted  by 
regulatory  authorities.  Those  metrics  and  curriculum  content  were  developed  by  only  one  or  two 
clinical  experts  whose  perspective  focused  entirely  upon  the  skill  or  procedure  to  be  taught,  and  not  on 
the  larger  needs  for  patient  safety,  which  is  a  major  concern  for  regulatory  authorities. 
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Currently  it  is  acknowledged  that  the  majority  of  robotic  surgery  is  practiced  on  the  da  Vinci 
robotic  surgical  system  (Intuitive  Surgical  Inc,  Sunnyvale,  CA).  As  a  result,  most  surgeons'  experience, 
and  immediate  future  expectation  for  clinical  practice,  is  with  that  device.  While  there  may  be  the 
perception  that  the  FRS  is  focused  upon  this  particular  robotic  surgery  system,  this  is  not  the  case. 
There  are  a  number  of  basic  principles  that  apply  to  any  robotic  system  that  will  be  developed  -  for 
example,  all  robots  must  be  initialized  (set  up,  adjust  scaling,  etc),  must  be  docked  (either  next  to  the 
patient,  above  the  patient  or  attached  to  the  table),  use  a  clutching  mechanism  to  navigate  the 
operative  field  with  multiple  instruments),  perform  specific  tasks  or  activities  (regardless  of  the  input 
device  or  end  effectors),  have  communication  with  other  team  members  (team  training)  and  safely 
undock  the  system.  To  that  end,  the  outcomes  measures  were  designed  to  be  device  agnostic  and  it  is 
with  appreciation  that  the  decades  of  developing  generic  flight  simulation  by  the  DoD  contributed  to  the 
approach  and  design  of  FRS.  Flowever,  the  working  group  recognized  the  widespread  adoption  of  a 
specific  system  to  date  and  made  conscious  decisions  to  avoid  proprietary  specifications  while 
acknowledging  the  importance  that  the  curriculum  be  capable  of  evaluating  the  skills  that  are  currently 
being  used  in  the  operating  room. 

Several  members  of  the  consensus  group  are  aware  of  the  development  of  surgical  robots  by 
other  companies  and  in  other  countries,  but  none  of  these  members  have  direct  experience  with  these 
devices  which  they  can  share  with  the  group.  Therefore,  attempts  to  address  the  operational 
characteristics  of  these  prototype  devices  would  be  purely  speculative  and  inappropriate  for  a  standard 
curriculum  in  robotics  at  this  time,  yet  it  was  felt  that  emerging  systems  will  need  to  address  the  tasks 
indicated  in  this  version  of  FRS.  The  evolving  nature  of  robotic  surgery  will  necessitate  future  revisions  of 
this  material  to  capture  the  variations  in  devices  and  procedures  that  will  emerge  in  the  future. 

Over  the  past  two  decades,  it  has  become  apparent  that  engagement  of  the  organizations  with 
the  appropriate  authority  for  standards  (e.g.  the  ACGME,  Residency  Review  Committees  (RRC),  the 
American  Board  of  Medical  Specialties  (ABMS))  , surgical  education  and  training  (the  numerous  surgical 
and  subspecialty  societies,  such  as  ACS,  SAGES,  AAGL,  AUA)  and  certification  (respective  surgical  and 
specialty  Boards)  are  critical  to  the  development  of  a  curriculum  that  is  meaningful  and  acceptable  to  all 
individuals  and  organizations  involved  in  the  "full  life  cycle"  of  training  .  The  example  in  Figure  3  of  one 
current  model  for  such  a  full  life-cycle  curriculum  demonstrates  that  the  initial  step  for  any  curriculum  is 
to  establish  the  appropriate  Outcomes  Measures.  It  should  be  noted  that  this  life  cycle  process  requires 
that  there  be  continuous  long-term  feedback  from  the  regulatory  and  certification  authorities  such  that 
iterative  improvement  of  the  curriculum  can  be  achieved  over  many  years,  referred  to  as  the 
'longitudinal  maintenance  of  a  curriculum'  over  time.  It  is  acknowledged  that  all  stakeholders  may  or 
may  not  initially  agree  to  require  such  a  curriculum.  Flowever  their  input  at  this  time  is  essential  to 
ensure  that,  if  over  time,  any  organization  would  want  to  reconsider  and  require  such  a  curriculum,  that 
organization  would  have  had  input  into  the  creation  of  the  curriculum.  It  is  evident  that  participation  by 
multiple  specialties  provides  the  essential  broad  perspective  that  could  create  a  stronger  curriculum, 
whether  it  is  adopted  now  or  in  the  future. 

Our  approach  to  begin  the  curriculum  development  with  an  initial  outcomes  measures 
conference  was  inspired  by  the  ACGME  and  ABMS  establishing  the  six  competencies  before  beginning 
the  task  of  developing  the  educational  and  training  curriculum  that  will  be  needed  to  teach  the 
competencies.  It  is  noteworthy  that  the  above  ACGME  competencies  are  inclusive  of  all  medical 
specialties.  In  a  similar  fashion,  this  FRS  curriculum  is  intended  to  serve  most  all  surgical  and  procedural 
specialties  that  currently  use  or  have  the  future  potential  to  perform  robotic  and  computer  assisted 
interventions.  It  is  anticipated  that,  if  created  within  a  framework  of  a  more  comprehensive  utilization, 
the  process  and  perhaps  the  curriculum  template  will  be  adopted  by  the  participating  specialties  to 
develop  their  own  "specialty  unique"  fundamentals  of  robotic  surgery.  This  concept  of  an  initial 
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template  upon  which  subsequent  curricula  can  be  easily  developed  has  been  proposed  by  Dr.  Robert 
Sweet  as  noted  above  (Figure  2).  The  advantages  of  using  a  common  template  are  twofold: 

1.  Comparative  Effectiveness  Analysis  -  Across  specialties.  A  common  process  would  permit  a 
more  scholarly  and  scientifically  valid  way  of  performing  comparative  effectiveness  analysis 
of  outcomes  for  the  identical  or  similar  robotic  procedures,  especially  if  two  specialties 
perform  the  same  procedure.  In  addition,  some  of  the  most  basic  types  of  procedural  skills 
(such  as  open,  laparoscopic,  flexible  endoscopic,  and  image  guided)  can  also  be  developed 
and  adopted  with  a  uniform  methodology,  as  the  FRS  has  the  potential  to  do,  thus  saving 
resources  by  eliminating  the  need  for  every  society  to  develop  their  own  variation  of  skills. 
This  also  provides  a  'de  facto  standard'  that  could  be  applied  towards  a  more  uniform  way 
of  developing  curricula. 

2.  Comparative  Effectiveness  Analysis  -  Within  a  specialty.  Such  a  common  process  would 
allow  much  easier  development  of  subsequent,  more-complicated,  specialty  curricula,  as 
indicated  by  the  Sweet  Tree.  It  will  also  be  possible  to  conduct  a  more  scientific  validation 
of  a  comparative  effectiveness  analysis  of  the  same  procedure  using  different  techniques 
(such  as  robotic  versus  laparoscopic  versus  open  surgery). 

This  methodology  has  been  successfully  executed  within  other  non-medical  simulation 
environments,  and  has  enabled  sharing  of  basic  elements  of  curriculum  development,  validation,  and 
certification.  This  has  resulted  in  saving  time  and  resources  by  eliminating  competition  and  redundancy. 
The  last  point  warrants  emphasis,  because  simulation  is  new  to  healthcare,  and  much  can  be  learned 
about  simulation  and  curriculum  development  from  the  more  than  eighty  years  of  experience  in  the 
aviation,  military,  and  nuclear  domains.  While  each  of  these  domains  is  unique,  it  is  the  process  and 
methodology  that  has  allowed  for  a  much  more  efficient  development  of  training  curricula.  As  a  first 
initiative,  it  is  anticipated  that  constructive  changes  will  occur  in  this  curriculum  as  feedback  is  used  to 
improve  this  initial  effort. 

Material  developed  under  FRS  in  this  work  focused  on  measuring  the  most  basic  skills  that  a 
surgeon  must  possess  in  order  to  perform  robotic  surgery.  Although  some  of  these  skills  require  a 
background  of  general  surgical  knowledge,  most  measures  of  competency  in  FRS  focus  on  cognitive  and 
psychomotor  technical  skills.  The  scope  was  limited  to  actions  performed  by  the  surgeon  in  preparing, 
performing,  and  finishing  a  robotic  procedure.  These  have  included  the  most  common  errors  that  are 
committed  in  each  of  these  areas.  The  actions  of  the  entire  surgical  team  were  not  part  of  this 
evaluation,  though  team  leadership  and  performance  were  recognized  as  critical.  The  surgeon's  role 
within  that  team  was  included. 

Conclusions:  The  consensus  conference  involving  members  from  major  stakeholder  organizations  in 
surgical  training,  governance,  and  certification  across  multiple  specialties  was  conducted  to  arrive  at  a 
consensus  regarding  the  most  important  outcome  measures  for  the  safe  conduct  of  robotic  surgery.  It  is 
anticipated  that  the  results  of  the  first  FRS  consensus  conference  will  be  used  by  the  Curriculum 
Development  Consensus  Conference  and  will  have  iterative  improvement  by  clinicians  that  will  be 
attending  that  conference.  The  development  of  FRS  is  multi-specialty,  system  agnostic  and  follows 
decades  of  experience  in  other  industries  at  developing  such  basic  education  and  training  platforms. 
Using  the  curriculum  for  training  and  assessment  should  result  in  a  surgeon  who  has  proficiency  in  basic 
robotic  surgery  skills  and  is  capable  of  passing  the  requirements  of  high  stakes  testing  and  evaluation. 
This  testing  and  evaluation  would  be  conducted  by  an  appropriate  independent,  objective  and 
authoritative  organization,  which  will  adopt  the  materials  developed  from  this  consensus  process. 


15 


141 


FRS  Consensus  Conference  I:  Outcomes  Measures 


Manuscript  Draft,  June  16,  2012 


Acknowledgements 

This  work  was  partially  sponsored  by  the  Department  of  the  Army,  Award  Number  W81XWH-11-2-0158 
to  the  recipient  Adventist  Health  System/Sunbelt,  Inc.,  Florida  Hospital  Nicholson  Center.  "The  U.S. 

Army  Medical  Research  Acquisition  Activity,  820  Chandler  Street,  Fort  Detrick  MD  21702-5014  is  the 
awarding  and  administering  acquisition  office."  The  content  of  the  information  does  not  necessarily 
reflect  the  position  or  the  policy  of  the  Government,  and  no  official  endorsement  should  be  inferred.  For 
purposes  of  this  article,  information  includes  news  releases,  articles,  manuscripts,  brochures, 
advertisements,  still  and  motion  pictures,  speeches,  trade  association  proceedings,  etc.  The  U.S.  Army 
Medical  Research  and  Materiel  Command  (USAMRMC)  shall  be  notified  by  recipient  prior  to  release  to 
the  public  of  planned  news  releases,  planned  publicity,  advertising  material  concerning 
grant/cooperative  agreement  work,  and  planned  presentations  to  scientific  meetings. 


REFERENCES 

1.  Adamsen  S,  Hansen  OH,  Funch-Jensen  P,  et  al.  Bile  duct  injury  during  laparoscopic  cholecystectomy: 
a  prospective  nationwide  series.  J  Am  CollSurg  1997;  184:  571-578. 

2.  Merrie  AE,  Booth  MW,  Shah  A,  et  al.  Bile  duct  imaging  and  injury:  a  regional  audit  of  laparoscopic 
cholecystectomy.  Aust  NZ  J  Surg  1997;  67:  706-711. 

3.  Shea  JA,  Berlin  JA,  Bachwich  DR,  et  al.  Indications  for  outcomes  of  cholecystectomy:  a  comparison  of 
the  pre  and  postlaparoscopic  eras.  Ann  Surg  1998;  227:  343-350. 

4.  Available  online  at 

http://www.acgme.org/acWebsite/downloads/RRC_progReq/440_general_surgery_01012008_u08 
102008.pdf  as  accessed  on  02/27/2012 

5.  Peters  JH,  Fried  GM,  Swanstrom  LL,  Soper  NJ,  Sillin  LF,  Schirmer  B,  Hoffman  K;  SAGES  FLS 
Committee.  Development  and  validation  of  a  comprehensive  program  of  education  and  assessment 
of  the  basic  fundamentals  of  laparoscopic  surgery.  Surgery.  2004  Jan;135(l):21-7. 

6.  Available  online  at  http://www.absurgery.org/default.jsp7news_newreqs  as  accessed  on  2/27/2012 

7.  Zorn  KC,  Gautam  G,  Shalhav  AL  et  al.  Training,  credentialing,  proctoring  and  medicolegal  risks  of 
robotic  urological  surgery:  recommendations  of  the  society  of  urologic  robotic  surgeons.  J  Urol. 

2009  Sep;  182(3):  1126-3 2. 

8.  Goh  AC,  Goldfarb  DW,  Sander  JC,  Miles  BJ,  Dunkin  BJ.  Global  evaluative  assessment  of  robotic  skills: 
validation  of  a  clinical  assessment  tool  to  measure  robotic  surgical  skills.  J  Urol.  2012 
Jan;187(l):247-52 

9.  Hung  AJ,  Patil  MB,  Zehnder  P,  Cai  J,  Ng  CK,  Aron  M,  Gill  IS,  Desai  MM.  Concurrent  and  predictive 
validation  of  a  novel  robotic  surgery  simulator:  a  prospective,  randomized  study.  J  Urol.  2012 
Feb;187(2):630-7. 

10.  Sweet  RM,  Hananel  D,  Lawrenz  F.  A  unified  approach  to  validation,  reliability,  and  education  study 
design  for  surgical  technical  skills  training.  Arch  Surg.  2010  Feb;145(2):197-201. 

11.  Dalkey  N.  The  Delphi  Method:  An  experimental  study  of  group  opinion.  RAND  Corporation 
Memorandum  (RM-5888-PR),  1969. 


16 


142 


FRS  Consensus  Conference  I:  Outcomes  Measures 


Manuscript  Draft,  June  16,  2012 


12.  Dulan  G,  Rege  RV,  Hogg  DC,  Gil berg-Fisher  KM,  Arain  NA,  Tesfay  ST,  Scott  DJ.  Developing  a 
comprehensive,  proficiency-based  training  program  for  robotic  surgery.  Surgery  (in  press). 

13.  Dulan  G,  Rege  RV,  Hogg  DC,  Gilberg-Fisher  KM,  Tesfay  ST,  Scott  DJ.  Content  and  face  validity  of  a 
comprehensive  robotic  skills  training  program  for  general  surgery,  urology,  and  gynecology.  Am  J 
Surg  (in  press). 


17 


143 


ACS-AEI  Meeting,  Chicago,  IL,  March  26,  2012 


Title 

Report  on  the  First  Consensus  Conference  on  the  Fundamentals  of  Robotic  Surgery:  Outcome 
Measures 

Authors 

R.M.  Satava,  University  of  Washington  School  of  Medicine,  Seattle,  WA 
R.D.  Smith,  Florida  Hospital  Nicholson  Center,  Celebration,  FL 
V.R.  Patel,  Global  Robotics  Institute,  Florida  Hospital,  Celebration,  FL 


Abstract 


FRS  Mission  Statement: 

“Create  and  develop  a  validated  multi-specialty,  technical  skills  competency  based 
curriculum  for  surgeons  to  safely  and  efficiently  perform  basic  robotic-assisted  surgery.” 


Purpose :  On  12-13  December,  2011  the  Fundamentals  of  Robotic  Surgery  (FRS)  Consensus 
Conference  (FRSCC#1)  on  Outcomes  Measures  convened  an  international  body  of  leaders  in 
robotic  surgery  to  define  the  skills  necessary  to  begin  the  process  of  creating  a  certifiable 
curriculum  and  testing  method  in  robotic  surgery. 

Goals :  To  identify  the  outcomes  that  must  be  measured  to  certify  that  a  surgeon  has  the  most 
basic  of  cognitive  and  psychomotor  technical  skills  for  robotic  surgery.  These  outcomes  are 
organized  as  a  list  of  tasks  that  a  surgeon  must  be  able  to  perform  successfully,  a  list  of  the  most 
common  errors  associated  with  each  task,  and  the  metrics  that  will  be  used  to  measure 
competency  in  that  task. 

Objectives'.  To  develop  a  list  of  skills,  tasks  and  errors  critical  to  the  performance  of  robotic 
surgery,  and  identify  quantitative  outcome  metrics  that  accurately  measure  perfonnance. 

Scope :  Material  developed  under  FRS  in  this  work  focused  on  measuring  the  most  basic  skills 
that  a  surgeon  must  possess  in  order  to  perform  robotic  surgery.  Although  some  of  these  skills 
require  a  background  of  general  surgical  knowledge,  most  measures  of  competency  in  FRS  were 
technical  (both  cognitive  and  psychomotor)  skills  specifically  required  and  essential  to  robotic 
surgery. 

The  scope  was  limited  to  actions  performed  by  the  surgeon  in  preparing,  performing,  and  after 
finishing  a  robotic  procedure  as  well  as  the  more  common  errors  in  each  of  these  areas.  The 
actions  of  the  entire  surgical  team  were  not  part  of  this  evaluation,  though  team  leadership  and 
performance  were  recognized  as  critical.  The  surgeon’s  role  within  that  team  was  included. 

Methodology.  The  Consensus  Conference  was  conducted  during  a  2  day  period  using  a 
modified  Delphi  methodology.  The  participants  consisted  of  subject  matter  experts  from  14 
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different  surgical  specialties  that  use  robotic  surgery,  as  well  as  representatives  from  a  number  of 
the  certifying  surgical  specialty  boards  and  surgical  education  societies,  and  included 
participation  by  the  civilian,  the  Department  of  Defense  and  the  Veterans  Administration  (VA) 
sectors.  Many  of  the  participants  are  members  of  the  ACS-AEI  and  of  the  Alliance  of  Surgical 
Specialties  for  Education  and  Training  (ASSET).  After  the  evaluation  of  existing  materials  and 
curricula,  a  task  deconstruction  was  perfonned  to  identify  the  tasks,  subtasks  and  errors  that  need 
to  be  measured.  A  matrix  was  then  created  that  matched  metrics  to  the  tasks,  skills  and  errors. 

Following  the  conference,  a  second  round  classic  Delphi  anonymous  rating  was  used  to  insure 
concurrence,  to  prioritize  the  ranking  of  the  tasks  and  to  eliminate  low-scoring  tasks. 

Results  :  The  results  provide  a  matrix  of  specific  robotic  surgery  tasks  that  are  matched  to  their 
common  errors,  a  description  of  the  desired  outcome  and  the  quantitative  metrics  that  support 
those  outcomes.  These  tasks  are  the  core  material  that  will  be  presented  at  this  meeting. 

Future  Directions:  The  measures  that  are  the  results  this  conference  will  be  utilized  as  the 
requirements  for  metrics  that  must  be  incorporated  into  the  curriculum  development  at  the 
FRSCC#2  Curriculum  Development  conference.  Following  the  completed  curriculum,  there 
will  be  a  FRSCC#3  Validation  Study  Design  conference,  the  design  of  which  will  be  utilized  in 
the  multi-institution  Validation  Study. 

Upon  completion,  the  validated  curriculum  will  be  transitioned  to  the  Fundamentals  of 
Laparoscopic  Surgery  Committee  of  SAGES/ACS  to  develop  the  high-stakes  testing  and 
evaluation  and  eventually  submitted  to  appropriate  certifying  boards  for  consideration  of 
adoption. 


*  This  work  was  supported  by  an  unrestricted  educational  grant  through  the  Minimally  Invasive  Robotics  Association  from  Intuitive  Surgical 
Incorporated. 

**  This  effort  was  also  sponsored  by  the  Department  of  the  Army,  Award  Number  W81XWH-11-2-0158  to  the  recipient  Adventist  Health 
System/Sunbelt,  Inc.,  Florida  Hospital  Nicholson  Center.  "The  U.S.  Army  Medical  Research  Acquisition  Activity,  820  Chandler  Street,  Fort  Detrick 
MD  21702-5014  is  the  awarding  and  administering  acquisition  office."  The  content  of  the  information  does  not  necessarily  reflect  the  position 
or  the  policy  of  the  Government,  and  no  official  endorsement  should  be  inferred.  For  purposes  of  this  article,  information  includes  news 
releases,  articles,  manuscripts,  brochures,  advertisements,  still  and  motion  pictures,  speeches,  trade  association  proceedings,  etc.  The  U.S. 
Army  Medical  Research  and  Materiel  Command  (USAMRMC)  shall  be  notified  by  recipient  prior  to  release  to  the  public  of  planned  news 
releases,  planned  publicity,  advertising  material  concerning  grant/cooperative  agreement  work,  and  planned  presentations  to  scientific 
meetings. 
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Abstract 
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Conference  (FRSCC#1)  on  Outcomes  Measures  convened  an  international  body  of  leaders  in 
robotic  surgery  to  define  the  skills  necessary  to  begin  the  process  of  creating  a  certifiable 
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Objectives'.  To  develop  a  list  of  skills,  tasks  and  errors  critical  to  the  performance  of  robotic 
surgery,  and  identify  quantitative  outcome  metrics  that  accurately  measure  perfonnance. 

Scope :  Material  developed  under  FRS  in  this  work  focused  on  measuring  the  most  basic  skills 
that  a  surgeon  must  possess  in  order  to  perform  robotic  surgery.  Although  some  of  these  skills 
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The  scope  was  limited  to  actions  performed  by  the  surgeon  in  preparing,  performing,  and  after 
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actions  of  the  entire  surgical  team  were  not  part  of  this  evaluation,  though  team  leadership  and 
performance  were  recognized  as  critical.  The  surgeon’s  role  within  that  team  was  included. 
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modified  Delphi  methodology.  The  participants  consisted  of  subject  matter  experts  from  14 
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different  surgical  specialties  that  use  robotic  surgery,  as  well  as  representatives  from  a  number  of 
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participation  by  the  civilian,  the  Department  of  Defense  and  the  Veterans  Administration  (VA) 
sectors.  Many  of  the  participants  are  members  of  the  ACS-AEI  and  of  the  Alliance  of  Surgical 
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Following  the  conference,  a  second  round  classic  Delphi  anonymous  rating  was  used  to  insure 
concurrence,  to  prioritize  the  ranking  of  the  tasks  and  to  eliminate  low-scoring  tasks. 

Results  :  The  results  provide  a  matrix  of  specific  robotic  surgery  tasks  that  are  matched  to  their 
common  errors,  a  description  of  the  desired  outcome  and  the  quantitative  metrics  that  support 
those  outcomes.  These  tasks  are  the  core  material  that  will  be  presented  at  this  meeting. 

Future  Directions:  The  measures  that  are  the  results  this  conference  will  be  utilized  as  the 
requirements  for  metrics  that  must  be  incorporated  into  the  curriculum  development  at  the 
FRSCC#2  Curriculum  Development  conference.  Following  the  completed  curriculum,  there 
will  be  a  FRSCC#3  Validation  Study  Design  conference,  the  design  of  which  will  be  utilized  in 
the  multi-institution  Validation  Study. 

Upon  completion,  the  validated  curriculum  will  be  transitioned  to  the  Fundamentals  of 
Laparoscopic  Surgery  Committee  of  SAGES/ACS  to  develop  the  high-stakes  testing  and 
evaluation  and  eventually  submitted  to  appropriate  certifying  boards  for  consideration  of 
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*  This  work  was  supported  by  an  unrestricted  educational  grant  through  the  Minimally  Invasive  Robotics  Association  from  Intuitive  Surgical 
Incorporated. 

**  This  effort  was  also  sponsored  by  the  Department  of  the  Army,  Award  Number  W81XWH-11-2-0158  to  the  recipient  Adventist  Health 
System/Sunbelt,  Inc.,  Florida  Hospital  Nicholson  Center.  "The  U.S.  Army  Medical  Research  Acquisition  Activity,  820  Chandler  Street,  Fort  Detrick 
MD  21702-5014  is  the  awarding  and  administering  acquisition  office."  The  content  of  the  information  does  not  necessarily  reflect  the  position 
or  the  policy  of  the  Government,  and  no  official  endorsement  should  be  inferred.  For  purposes  of  this  article,  information  includes  news 
releases,  articles,  manuscripts,  brochures,  advertisements,  still  and  motion  pictures,  speeches,  trade  association  proceedings,  etc.  The  U.S. 
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